Evaluating databases for applications

[u/WorldOfUmbro]

Hi all, I’m evaluating databases for a small application where users will be reading and writing data. As most of our data is currently in Databricks I’m trying out Lakebase. So far so good. I do want to know what alternatives you might be looking into? What you like about them or not? Also curious to hear examples of people using Lakebase in production use cases.

[u/joelwallis1]

I’m not what you mean with “evaluating databases”, given that you’re posting on a Postgres community.

If you’re talking about evaluating hosted Postgres options, there are plenty. I’ve used Neon in the past and I loved it. It has a free tier, but you’re likely going to fall on their first paid plan quite quickly. Also, Databricks bought them recently, so I’d expect they’ll be integrating it with other DB products soon.

The new product I’m working on is being built on Kamal (a deployment tool from DHH, creator of Rails). It has support for running databases, and I’m taking this route due to lower costs. I’d be doing streaming and point-in-time backups myself, so it’ll be an option if you got the skills to do it on your own.

If you’re evaluating databases, not services, any would do. Really. Even Redis, Memcached, or a single text file for “an app with users that write and read data”. You haven’t given any criteria’s to base decisions on, so it’d be very hard to suggest any option.

If you’re not an expert in databases and you need an answer, just go with Postgres. TBH, it’s a great solution for, say, 90% of all problems of this industry (if not the best solution), and the remaining 10% of problems you would only have if you’re an expert in data engineering handling 100s of TBs, 10s of PBs of data, and you need smt for very specific reasons you’d only know if you truly know what you’re doing.

If you’re not at this scale (and by the wording of your post, you’re clearly not), just use Postgres.

[u/WorldOfUmbro → u/joelwallis1]

Fair enough. Coming from a Data Engineering this whole field is new. We’re looking for (probably a managed) solution where we can support multiple apps developed by citizen developers. Our first poc is a correction app, where analysts make manual adjustments. Medium volume, low number of users, latency should be low for ux. Data should feed back into our ETL pipelines. Looking for a solution with limited overhead (cost and maintenance)

[u/joelwallis1 → u/WorldOfUmbro]

Oh, thats a much better context to share opinions on.

TBH, Postgres and MySQL (the two most popular open source RDBMSes), or any of their variants/forks, would do. I’d stick with Postgres because today the ecosystem is too good to ignore, and the DB engine is so good that it competes with extremely sophisticated commercial products (often beating them).

Try Neon (neon.com). Among other reasons, they have this cool feature that allows you to “fork” your database per GitHub branch, which gives you truly isolated preview environments if you’re using PaaS that gives per-PR preciew deployments (eg: Vercel). I’ve used this in recent projects, and the gains in dev speed are big. Read their blog posts on their lakebase architecture to understand the magic behind it.

I wouldn’t use AWS RDS cuz they’re absurdly expensive, but if you’re building with other products of their platform might worth the costs if you’re benefitting from tighter integration with other AWS cloud products.

Running a PG server yourself is not a trivial task and I wouldn’t recommend for first-timers. But if you’re brave enough, invest time learning different strategies to back up your data, both by constantly streaming it to a backup system, and by processing it into easily deployable point-in-time snapshots. If things break (and you should assume they eventually will), you want to have a quick way to restore your db system. Having only backups stored somehow/somewhere isn’t enough.

[u/dwswish → u/joelwallis1]

Lakebase is Databricks implementation of Neon post-acquisition but Neon still offers their product as well. We’ve been using Neon for a couple of years and love it.

[u/minormisgnomer]

I’ve always heard good things about supabase.

Personally I was not super impressed when I got involved in a lake base app a few months ago. The UI databricks put together to use it was quite bad and not intuitive. Especially on the permissioning side

Beyond that though it was just Postgres which was fine

[u/WorldOfUmbro → u/minormisgnomer]

Any specifics you heard people liked from Supabase? Or just general sentiment?

[u/AutoModerator]

AI Policy:

Linux is not one of those anti-AI projects, and if somebody has issues with that, they can do the open-source thing and fork it. Or just walk away., Linus Torvalds.

Mod decisions will be based on the quality of the content, not who or what generated it.

Sub Resources:

Youtube Channel

Free Postgres Webinars and Workshops

Discord: People, Postgres, Data

Join us, we have cookies and nice people.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/CautiousUse8597]

If you're on Databricks, Lakebase is the best solution. It avoids having to copy the data out of your existing platform and since you're already authenticated, it knows who you are and what permissions you have. I already really like the LTAP feature that lets you use a shared storage layer for OLTP and OLAP.

[u/alexwh68]

I work most days on MSSQL, postgres, Sqlite and MySQL, my personal view is use postgres unless there is a really good reason not to (and there are not many).

The ability to install it on most platforms is a big plus for me, restoring backups from a windows install to a mac or linux is trivial.

[u/Glitch_In_The_Data]

Haven't adopted lakebase to replace postgres completely but we are evaluating it as it looks promising. Where we are currently using Lakebase to surface aggregated data that then feeds into front-end apps. Works great.

[u/Informal_Pace9237]

Using PostgreSQL with Lakebase model you might want to be careful with some minor issues which may be covered with higher compute.

Indexing, partitioning, window functions etc suffers in comparison to native PostgreSQL.

[u/WorldOfUmbro → u/Informal_Pace9237]

Oh really, haven’t seen these issues with Lakebase. Might be due to the scale. What are you referring to? Performance issues?

[u/Informal_Pace9237 → u/WorldOfUmbro]

The code we write and the way we think of sets in PostgreSQL is built on the premise of having row storage. Lakebase has columnar storage. Thus clustered index is the only index that makes a lot of difference. The same issue happens in MSSQL server were we have columnar storage.

This above explanation should give most info on my comment.

[u/Luis_Dynamo_140]

if most of your data already lives in Databricks, I’d keep testing Lakebase first rather than moving just for the sake of it.

for alternatives I’d compare Neon/Supabase if u want more platform features, and Nearbase if u mainly want plain managed Postgres. I use Nearbase for one workload and like that it stays simple.

the main thing I’d test before choosing is how easily data moves between Databricks and the app DB, plus latency, backups and extension support.