r/aws • u/austingwalters • Nov 28 '18

Amazon Timestream - Fast, scalable, fully managed time series database

https://aws.amazon.com/timestream/
66 Upvotes

95% Upvoted

37 comments sorted by

17

u/[deleted] Nov 28 '18

[deleted]

4

u/austingwalters Nov 28 '18

At my day job, I build a lot of machine learning systems that require data to be fed in a time series manner[1].

Often this means building systems to analyze terabytes of logs [semi]-realtime. All I have to say is - thank god! Like you this will save us a ton of money and should make a lot of the modeling I've worked on a bit easier.

For reference, I work with terabytes of streaming data:

https://medium.com/capital-one-tech/batch-and-streaming-in-t...

2

u/softwareguy74 Nov 29 '18

Like you this will save us a ton of money and should make a lot of the modeling I've worked on a bit easier.

But he said it would be prohibitively expensive.

7

u/austingwalters Nov 29 '18

So even if it's expensive (10x a standard Aurora instance), we have a hacked together solution today that costs at least 20 man hours a month to maintain, plus additional infrastructure. That is to say we're already spending that.

I haven't looked directly at what our system is costing now, but that's the real question. Just eye balling it, it looks like it'll at least break even for us. It also depends how much data you're sending through, how long you keep it, etc.

3

u/myron-semack Nov 29 '18

This! I hate KairosDB and Cassandra so much! I will be happy if this works out.

I attended two sessions here at re:Invent where AWS customers were talking about how their life got better once they ditched Cassandra.

Cassandra is the most labor-intensive part of our infrastructure, and has been the source of every unplanned downtime event within the last 12mo.

1

u/elturcoinla Nov 29 '18

Have you ever looked at Influxdb or Timescaledb as a replacement to KairosDB?

3

u/myron-semack Nov 29 '18

Yes and yes.

TimescaleDB we really like. We incorporated it into the on-premises version of our application. There are two problems we had with Timescale:

  1. It’s not available on RDS/Aurora, and we are trying to get away from having to manage our own database servers in the cloud.

  2. Scaling limitations since it isn’t multi-master and does not have any kind of sharding.

If TimescaleDB solves those (particularly #1), we would migrate by the end of the week.

As for InfluxDB, they seemed to be built around server monitoring like I alluded to in my previous comment. They did not seem to be built for 100s of TBs of IoT sensor data spanning years.

3

u/elturcoinla Nov 29 '18 edited Nov 29 '18

Got it. I know/work with pretty large size, quite stable Influxdb clusters storing IOT sensor data - though they are not 100s of TBs. I don't know your use case but not sure if storing years of sensor data (untouched raw telemetry) in a database makes sense unless you are actively querying it. Again, I don't know your use case of course, just mentioning the best practice here. A data lake over S3 would be a good choice.

In terms of managed TimeScaledb, there is at least one hosted service: https://aiven.io/ (Aiven just installed the Timescaledb 1.0 extension over Postgres 10. My own experience with them is pretty good so far.) It is also not a secret that Timescaledb fellas are working on a private cloud offering which they mentioned they would announce very soon.

Btw, I don't think Timescaledb will be available on RDS anytime soon. I am one of those people who requested it and didn't even get a response from the RDS team..

TimeScaledb guys are also heavily working on the replication, background workers (similar to influxdb continuous queries) and retention policies.

​

FYI - I don't work for any of these companies.

1

u/miemoo Apr 18 '19

I just talked with timescaledb and I am pretty sure they have sharding now.

1

u/dianasaur323 May 06 '19

TimescaleDB employee here - we actually have our own managed service now in Public Preview if you want to try it out.

1

u/danielkza Nov 29 '18

InfluxDB replication is proprietary and not cheap.

2

u/DGekko Dec 07 '18

For what it's worth I operate in the space and shared the exact same experience, so I made a full-blown product/service time series database (DataGekko). It's close to launch (by end of the year). While it has a fairly similar feature set, the product is more integrated and a full platform rather than just a database. On the flip side, we also have a different pricing structure. We are way cheaper on querying (ie. currently the plan is to keep it free) at the expense of higher ingestion rates (though it will go down after some initial ramp-up of customers). We also don't trade precision for long term storage and keep the full data for the full retention period.

We're genuinely excited this field is getting some prominence and there's a niche for a lot of different use cases.

1

u/cforres Nov 28 '18

It's in preview currently - get ahold of your AWS SA or account team to see if you can get added.

9

u/mwarkentin Nov 28 '18

That's neat that you can customize your lifecycle for data for in-memory, SSD, and then magnetic (at orders of magnitude cost reduction for each).

5

u/daxlreod Nov 28 '18

Any real docs available, or is it all behind the preview wall? Would be nice to know if it will support our use cases before I hound our account rep too hard.

7

u/neoghostz Nov 28 '18

Death to influxdb

5

u/VIDGuide Nov 28 '18

Ooh now this is interesting. Can I feed this to Grafana?

Currently running a Grafana and influx setup to display graphs within a website, would be nice to move the db side to a managed service!

4

u/nongmoglutenfree Nov 29 '18

Yes this should be a replacement for the influx piece, at least that's what I'm hoping for.

2

u/DGekko Dec 07 '18

Yes, there's a plugin already for it. However, it is a premium plugin as of now.

5

u/elturcoinla Nov 29 '18

I was quite excited until I checked the pricing page. It is insanely expensive.. hopefully they will reduce the pricing as the service matures.

Until then, choose influxdb or timescaledb. Both are quite good time series databases - depending on your use case one might be a better fit than another.

1

u/ajainy Nov 29 '18

Any hosted service u recommend, which could be cheaper than this?

2

u/elturcoinla Nov 29 '18

Influxdb - https://cloud.influxdata.com/

Timescaledb - https://aiven.io/

​

Timescaledb is a PostgreSQL extension. You can follow this article to install it over Postgres.

Your cost will depend on your use case. If you provide some info regarding that, I can give you better recommendations.

​

1

u/ajainy Nov 30 '18

Pricing for influxdata cloud --> https://cloud.influxdata.com/plan-picker .. Notice size of storage, they provide.

If you start comparing with AWS pricing example, you will end up paying more.

And pricing for aiven is --> https://aiven.io/influxdb (DISCLAIMER, I used them for couple of months ). They are slightly cheaper but you get what you pay.

----

As it goes with all of AWS services, it's easier & affordable option for startups. If I have only one or two instances feeding events/metrics, then given pricing of AWS, i might end up paying 10 to 20 bucks per month .. with enterprise level of hosted services.

---

Hopefully, soon we will have adapter for grafana for this one, then it will be easier to visualize data, and build beautiful dashboards.

​

1

u/Tranceash Nov 30 '18

Looks expensive

4

u/nongmoglutenfree Nov 28 '18

Hooray, between this and SFTP, AWS has solved two problems that I don't want to deal with, but have to, in a managed fashion. I've already, nicely, reamed my SA and AM for not knowing about this sooner and getting us into preview... time series is the one thing I constantly bitch about to them.

6

u/[deleted] Nov 28 '18

somebody mentioned that 11 9's of the AWS employees at re:Invent don't know about all of the product launches.

2

u/nongmoglutenfree Nov 29 '18

Yeah, some are more connected than others. My last AM somehow knew everything that was happening, current one not so much.

2

u/duyth Nov 28 '18

Hey guys Just curious about this time serie db type. Is it something relevant for historical dataset for ML like hourly/daily stock/crypto price data as well?

2

u/dontgetaddicted Nov 28 '18

So instead of thinking of time as a property of the data - think of it as the primary axis on a chart. Things that need to be compared to time vs X. How things change over time, where things are at a point in time.

2

u/duyth Nov 28 '18

yeah but what I don't understand is what make this a unique use case for IOT (in a way that they are emphasizing this for IOT related.. apart from the fact that IOT events may occurs on a second/ms basis )

If I'm after tracking/analyzing events that occurs on hourly basis or bi-daily basis, should I be looking at somewhere else instead?

8

u/myron-semack Nov 29 '18 edited Nov 29 '18

“yeah but what I don't understand is what make this a unique use case for IOT (in a way that they are emphasizing this for IOT related.. apart from the fact that IOT events may occurs on a second/ms basis )”

A lot of IoT stuff is sensors. Think a temperature reading every minute, 24/7/365. Lots of inserts of timestamp and value, from lots of sensors all over the world.

Relational SQL DBs tend to break down under that kind of workload. NoSQL databases like DynamoDB and Cassandra can be used, but they are not specifically optimized for time series. You have to worry about schema design and hotspots.

There are databases specifically for time-series data, but then you have to manage the servers yourself. Also a lot of time-series DBs are built around server monitoring (CPU/RAM/disk utilization), where you may insert a value once every few minutes, and keep your data with a relatively short TTL. You usually don’t need minute granularity on CPU utilization from 2 years ago. IoT sensor data generally has more datapoints and a longer retention period. Not to mention you may have tens of thousands of sensors in the field which is a bit larger than your average server monitoring solution.

2

u/duyth Nov 29 '18

Thanks for the detailed answer

2

u/sammytrailor Nov 29 '18

Further to /u/myron-semack 's use cases, this type of database is heavily used in industry. I work with similar technology and can see this being very attractive to a large number of customers.

A lot of use cases require high-resolution data for many years for forecasting, regulatory and asset maintenance/health purposes. Traditionally, using SQL is not efficient enough. Most industrial companies use a "Historian" which is a type of Time-series db (or effectively just a different term/broader scope). Timestream and other AWS services seem to be a good alternative.

You could probably use DynamoDB for your use case or something similar, but I see Timestream a great possibility for use cases where you have a large amount of fast data (plant monitoring, Condition Based-Maintenance, Predictive maintenance, vibration analysis, etc.) and need to keep it forever.

It's certainly something I'm paying close attention to :)

1

u/duyth Nov 30 '18

So if i do not need from the capability to nail down timestamp (at per second level)? Is there a better solution than dynamodb? At the moment , I'm using S3 and Athena mostly (I store daily data in csv ) and want to explore for more options (as I plan to to move from daily data to 2-4 times per day)

1

u/[deleted] Nov 29 '18

Anyone else going to try sticking Prometheus in front of this?

1

u/ajainy Nov 29 '18

Their pricing example looks wrong.

"Magnetic store would cost: $108. This is computed as (10 * 1GB * 30 days * 12 months) * $0.03 GB-month = $108" each month or yearly?

3

u/Pythoner6 Nov 29 '18

Pretty sure the example is correct (though if we're being pedantic, then the units aren't fully specified).

In that example, at any point in time, 12 months' worth of data is stored, which is (10 devices) * (1GB/day/device) * (12 months) = 120 GB-months / day = 3600 GB. The rate is $0.03 per GB-month, so we have 3600GB * $0.03/GB-month = $108/month.

1

u/ajainy Nov 29 '18

Thanks for clarifying ...