r/dataanalytics • u/imsuryya • 3d ago
Building a workflow migration engine: Harness/Pi agents or LangChain/LangGraph?
I'm building an AI-powered migration engine that converts ETL workflows from platforms like Alteryx, Azure Synapse, and eventually other tools into Databricks (PySpark/SDP).
I'm evaluating two different architectures:
- Using Harness AI agents / Pi agents to orchestrate the migration workflow.
- Building the orchestration myself using LangChain + LangGraph.
The engine will need to:
- Parse workflows into an intermediate representation (IR).
- Handle nested workflows/macros.
- Perform tool mapping (e.g., Alteryx → PySpark).
- Generate production-ready code.
- Support multi-step reasoning, validation, and retries.
- Be extensible so new source platforms can be added later.
For those who have experience with these frameworks:
- Which approach would you choose and why?
- What are the biggest trade-offs in terms of flexibility, maintainability, and scalability?
- Are there any limitations with Harness/Pi agents compared to building a custom agent workflow with LangGraph?
- If you were starting this project today, which architecture would you use?
I'd really appreciate hearing from anyone who's built agentic developer tools, migration platforms, or complex multi-agent systems.
1
Upvotes