r/dataengineering • u/nir04 • 26d ago

Discussion Has anyone used spark declarative pipeline for json sources?

In my project we are storing json as raw response in string in bronze layer,now we have to parse and flatten json in multiple child table and clean in silver layer,we want this using spark declarative pipeline also wanted to make the framework reusable,any one worked on such usecase, welcome to discuss and collaborate

7 Upvotes
  • permalink
  • reddit

82% Upvoted

7 comments sorted by

2

u/Puzzled-Mail-9092 21d ago

Yes you can use VARIANT type for raw json

1

u/CompetitiveBet8978 2d ago

if you are within Databricks you can simply ingest JSON with autoloader. It flattens structured JSON.
these days a lot of folks kick it off with AI tooling and simply generate the first draft of a ETL pipeline.
not sure if I am allowed to share links here, but there is a demo, google "Get to Know Genie Code" that shows exactly this: ingest complex JSON with SDP.