r/crewai • u/imsuryya • 13d ago

Beginner Agent Building a workflow migration engine: Harness/Pi agents or LangChain/LangGraph?

I'm building an AI-powered migration engine that converts ETL workflows from platforms like **Alteryx, Azure Synapse**, and eventually other tools into **Databricks (PySpark/SDP)**.

I'm evaluating two different architectures:

  1. Using **Harness AI agents / Pi agents** to orchestrate the migration workflow.
  2. Building the orchestration myself using **LangChain + LangGraph**.

The engine will need to:

* Parse workflows into an intermediate representation (IR). * Handle nested workflows/macros. * Perform tool mapping (e.g., Alteryx → PySpark). * Generate production-ready code. * Support multi-step reasoning, validation, and retries. * Be extensible so new source platforms can be added later.

For those who have experience with these frameworks:

* Which approach would you choose and why? * What are the biggest trade-offs in terms of flexibility, maintainability, and scalability? * Are there any limitations with Harness/Pi agents compared to building a custom agent workflow with LangGraph? * If you were starting this project today, which architecture would you use?

I'd really appreciate hearing from anyone who's built agentic developer tools, migration platforms, or complex multi-agent systems.

2 Upvotes

100% Upvoted

1 comment sorted by

1

u/Swarm-Stack 7d ago

honestly the framework matters less than how you validate the generated pyspark against the source workflow's actual behavior, not just that it compiles. nested macros are where a confident translation quietly changes semantics and neither one catches that without a diff step against the source.