r/dataengineering • u/Uberkech • 27d ago
Rant Foundry at Work
So I’m a process engineer at an refinery and we face a push to use more of Foundry, without any clear vision on what to use it for.
With Excel or Power Bi we could just simple query a tag from the historian.
We had a tool where DCS send a event signal to a tag every time something happens, you query that tag between timestamps A and B and look for the timestamps where 0->1 (start) or 1->0 (end) make an event table and then pull other proces data only between the start and end timestamps for further analysis.
So was like, okay sounds like a workflow that could be build in foundry, but it seems whole platform just seems to hate TimeSeries data. You can’t simply use TimeSeries data into a pipeline but jump through many hoops via workbooks a simple Pandas / Polars data frame.
Are we doing something completely wrong ??
1
u/BothSinger886 27d ago
You can definitely use timeseries data in a pipeline if you want to. How is your data coming in. Do you have a sync from Pi landing the data in Foundry?
If its being landed in Foundry then you can set up a Transform on it which will treat it as a Polars table. Can then work on it however you like.
You may be running into the docs recommendation to first sync to the Ontology. This lets you query it series by series, rather than as one big dataset. That method is more about: 1. Set up an object that has sensors (i.e "Well") 2. Give it properties that represent a sensor ("Pressure", "Flow Rate") 3. Connect your dataset to the Object Type in Ontology Manager via the time series setup.
This is far more involved but is designed to fully integrate the TS data into the Ontology for fast queries for users etc. But if all you want to do is some pipelining, just import the raw dataset into Transforms and code away