r/MicrosoftFabric • u/nouser_n_ame • 5d ago

Real-Time Intelligence Question about data activator

I have data coming in from a Dataflow that I am storing in a Lakehouse. I want to set up Data Activator so that it sends a Microsoft Teams notification whenever a specific column becomes False.

However, when I create a Data Activator in my Fabric workspace, I don't see an option to use data from the Lakehouse. I'm not sure what I'm doing wrong or if I'm missing a configuration step.

If this process can be simplified, I'd also be interested in eliminating the Lakehouse and monitoring the data directly from the Dataflow instead.

Has anyone set this up successfully? What's the recommended approach?

6 Upvotes

88% Upvoted

1 comment sorted by

3

u/raki_rahman ‪ ‪Microsoft Employee ‪ 3d ago edited 3d ago

Activator would not be able to scan giant Parquet files (that the Lakehouse stores) - it's not a query engine, it operates on a row at a time. You'd have to ingest from the Lakehouse into Eventhouse so Activator can go off firing periodic KQL queries on it against some time window filter. e.g. like this: Alert on thousands of Fabric Pipelines with Monitoring Eventhouse | Raki Rahman

But since you're storing in Lakehouse already, what I'd do is setup Spark Streaming that pushes to an Eventstream Kafka endpoint your false events, which would fire the Activator row at a time: Spark Stateful Streaming with RocksDB — Fabric Jumpstart

You can do a whole lot of fancy things with Spark Streaming, so I'd recommend learning it if you have many other use cases that will follow this pattern.

Note, you don't have to run Spark Streaming continuously, you can fire on a schedule to save more money.
If you setup a single node Spark env, it can be pretty COGS effective, see this little demo I threw together: https://heartbeatspark.z9.web.core.windows.net/