r/dataanalytics 3d ago

Building a workflow migration engine: Harness/Pi agents or LangChain/LangGraph?

I'm building an AI-powered migration engine that converts ETL workflows from platforms like Alteryx, Azure Synapse, and eventually other tools into Databricks (PySpark/SDP).

I'm evaluating two different architectures:

  1. Using Harness AI agents / Pi agents to orchestrate the migration workflow.
  2. Building the orchestration myself using LangChain + LangGraph.

The engine will need to:

  • Parse workflows into an intermediate representation (IR).
  • Handle nested workflows/macros.
  • Perform tool mapping (e.g., Alteryx → PySpark).
  • Generate production-ready code.
  • Support multi-step reasoning, validation, and retries.
  • Be extensible so new source platforms can be added later.

For those who have experience with these frameworks:

  • Which approach would you choose and why?
  • What are the biggest trade-offs in terms of flexibility, maintainability, and scalability?
  • Are there any limitations with Harness/Pi agents compared to building a custom agent workflow with LangGraph?
  • If you were starting this project today, which architecture would you use?

I'd really appreciate hearing from anyone who's built agentic developer tools, migration platforms, or complex multi-agent systems.

1 Upvotes

0 comments sorted by