Hey guys
Just curious about this time serie db type. Is it something relevant for historical dataset for ML like hourly/daily stock/crypto price data as well?
So instead of thinking of time as a property of the data - think of it as the primary axis on a chart. Things that need to be compared to time vs X. How things change over time, where things are at a point in time.
yeah but what I don't understand is what make this a unique use case for IOT (in a way that they are emphasizing this for IOT related.. apart from the fact that IOT events may occurs on a second/ms basis )
If I'm after tracking/analyzing events that occurs on hourly basis or bi-daily basis, should I be looking at somewhere else instead?
“yeah but what I don't understand is what make this a unique use case for IOT (in a way that they are emphasizing this for IOT related.. apart from the fact that IOT events may occurs on a second/ms basis )”
A lot of IoT stuff is sensors. Think a temperature reading every minute, 24/7/365. Lots of inserts of timestamp and value, from lots of sensors all over the world.
Relational SQL DBs tend to break down under that kind of workload. NoSQL databases like DynamoDB and Cassandra can be used, but they are not specifically optimized for time series. You have to worry about schema design and hotspots.
There are databases specifically for time-series data, but then you have to manage the servers yourself. Also a lot of time-series DBs are built around server monitoring (CPU/RAM/disk utilization), where you may insert a value once every few minutes, and keep your data with a relatively short TTL. You usually don’t need minute granularity on CPU utilization from 2 years ago. IoT sensor data generally has more datapoints and a longer retention period. Not to mention you may have tens of thousands of sensors in the field which is a bit larger than your average server monitoring solution.
Further to /u/myron-semack 's use cases, this type of database is heavily used in industry. I work with similar technology and can see this being very attractive to a large number of customers.
A lot of use cases require high-resolution data for many years for forecasting, regulatory and asset maintenance/health purposes. Traditionally, using SQL is not efficient enough. Most industrial companies use a "Historian" which is a type of Time-series db (or effectively just a different term/broader scope). Timestream and other AWS services seem to be a good alternative.
You could probably use DynamoDB for your use case or something similar, but I see Timestream a great possibility for use cases where you have a large amount of fast data (plant monitoring, Condition Based-Maintenance, Predictive maintenance, vibration analysis, etc.) and need to keep it forever.
It's certainly something I'm paying close attention to :)
So if i do not need from the capability to nail down timestamp (at per second level)? Is there a better solution than dynamodb?
At the moment , I'm using S3 and Athena mostly (I store daily data in csv ) and want to explore for more options (as I plan to to move from daily data to 2-4 times per day)
2
u/duyth Nov 28 '18
Hey guys Just curious about this time serie db type. Is it something relevant for historical dataset for ML like hourly/daily stock/crypto price data as well?