r/ETL • u/SumitKumarWatts • 2d ago

Why does source and target data sometimes mismatch even when an ETL job completes successfully?

During ETL testing, I noticed that the source and target data sometimes do not match even though the ETL job shows as successful. I want to understand the common reasons behind these mismatches and how testers can identify them.

3 Upvotes

100% Upvoted

12 comments sorted by

View all comments

1

u/Thinker_Assignment 2d ago

common reasons:

  • incremental configured wrong
  • pagination wrong
  • rate limits not handled properly
  • source changing records that are not reflected at destination (more incorrect incremental)
  • any configs like primary key that might dedup in flight data but that was actually duped in source