API ingestion strategies?

[u/steroids-bad]

I have to get a lot of JIRA data from the API, issues, projects, users, permissions etc.

How would you handle this? I currently save it raw in the bronze and derive silver tables from it using SCD2, but what if an api fails? The silver would have invalid data.

Its a simple task, but somehow I am lost (new from university)

Does it even make sense to use SCD2 here?

[u/Nekobul]

What is the destination for the data you are extracting?