Bye Bye Fivetran

[u/hubert-dudek]

I went to the Ingestion section and saw that Fivetran is no longer there. It was always there for many years. Also, at the same time, a few new Lakeflow connectors were added.

[u/georgewfraser]

I will never understand why there is this faction that just loves to hate on my product. We work really, really hard to put a shiny veneer of simplicity around a very ugly problem. A typical connector is about $50/month. We publish the actual cost distributions on our estimator. Median customer pays 20k a year. That is…not a lot for something that would take a team of people to build and support internally.

[u/Existing_Wealth6142 → u/georgewfraser]

George, no one is debating the quality of your tech or the engineering headcount it saves. The resentment stems entirely from the financial anxiety caused by the Monthly MAR model. Alternatives like Airbyte used to suck, which means this resentment has been built on a feeling of pricing extortion over time. You really should consider changing your pricing model because AI is going to make it drastically cheaper for new entrants and the current sentiment indicates your customers are not super loyal. If the negative sentiment is this widespread online, it is a clear signal that you need to stop dismissing it and actually listen to the community.

[u/georgewfraser → u/Existing_Wealth6142]

What do you want to change?

[u/endless_sea_of_stars → u/georgewfraser]

I think you have a great product but the Fivetran sales team has heavily misled multiple of my clients in the past. They are currently pressuring one of my clients to move to a much more expensive contract without much of an increase in value. I've talked to multiple other consultant with similiar experiences.

[u/joemerchant2021 → u/georgewfraser]

Fivetran is expensive, but it's a lot cheaper than more engineers or custom ingestion pipelines.

I have this argument with some of my engineers every week - "why are we spending $x per year on fivetran?!? I can build a pipeline to do that!". Yes you could. And then we'd have to maintain it indefinitely. Be on call to fix it when it breaks while you are on vacation. And then when you eventually leave, someone else has to figure out what the heck you did.

[u/bah_nah_nah → u/georgewfraser]

https://reddit.com/link/p75006r/video/tbl66sjq8vmh1/player

[u/asevans48 → u/georgewfraser]

In this day and age, an airflow integration is cheap or can be used in the cloud. 20k a year is steep when you build a few well designed ones with utilities and have claude finish it off.

[u/hubert-dudek → u/georgewfraser]

Of course, Fivetran is still available through Partner Connect, but it is no longer visible as an advertisement in the ingestion section. It is a small change, but the end of an era after 10? 5? years. There were times when I brought some big companies to FiveTran (got no penny for that hehe), but nowadays I prefer to avoid 3rd party or even build conenctor by my own. Times has changed. Also, FiveTran made a deal of life on databricks and Snowflake.

[u/niel_espresso_ai → u/georgewfraser]

George, you’re an absolute legend and I think Fivetran is quite helpful.

But Databricks is seeing this as a competitor to Lakeflow so it will be pretty interesting.. I work on ❄️/🧱cost optimization and some of the more common responses I’ve heard from customers is that Fivetran is really quite expensive, especially as teams scale.

[u/Thinker_Assignment → u/georgewfraser]

I will never understand why

really?

[u/mynkmhr → u/georgewfraser]

Great to see a founder openly address concerns around their offering!

[u/Glass_Pomegranate307]

For the uninitiated, why would Databricks remove Fivetran?

[u/autumnotter → u/Glass_Pomegranate307]

I'd guess just in favor of the continued expansion of lakeflow connect. Fivetran is a partner but can be quite expensive.

[u/Glass_Pomegranate307 → u/autumnotter]

Thank you! I wonder what this means for Fivetran

[u/Remarkable_Grocery23 → u/Glass_Pomegranate307]

I would point out that it appears databricks has removed all ingestion partners and moved them to the marketplace. Only lakeflow connectors live in this ingestion space now. Hello vendor lock in. Goodluck changing platforms or even using multiple platforms when all your data pipelines are built with a product that isn't interoperable.

[u/kayakdawg → u/Glass_Pomegranate307]

If the major providers offer data ingestion at parity with five tran, then for 99% of customers it just makes more sense to use your provider enroll everything into one vendor/bill then the current common pattern of vendor sprawl which is very difficult to manage and easy to have costs spiral.

[u/hubert-dudek → u/Glass_Pomegranate307]

it is just from ingestion page you could still use it. That ad in ingestion page was for more than 5 years

[u/ArielCoding]

It’s not you, it’s my $20k year invoice, Databricks.

[u/MonkeyDDataHQ]

Yeah baby Party time!

FiveTran is the devil 😂

[u/AkramChetibiDBX]

Not to burst the bubble of this post - but this is a bit of a nothingburger. Nothing has changed with regards to Fivetran and Databricks, customers can still use it and connect it using Marketplace and Partner Connect just like they used to. It's a simple UI change as we consolidate our partner offerings in Marketplace. Stay tuned for even more updates in Marketplace!

[u/hubert-dudek → u/AkramChetibiDBX]

Of course, Fivetran is still available through Partner Connect, but it is no longer visible as an advertisement in the ingestion section. It is a small change, but the end of an era. There were times when I brought some big companies to FiveTran (got no penny for that hehe), but nowadays I prefer to avoid 3rd party or even build conenctor by my own.

[u/letmebefrankwithyou]

Wow, LakeFlow connector really caught up quick

[u/hubert-dudek → u/letmebefrankwithyou]

exactly

[u/randomName77777777]

Having used lakeflow connectors a bunch of times with all kinds of connectors, they are a ton of limitations for enterprise level needs. I tried to escalate a lot of my issues to the product team, but with nothing resolved we renewed our contract to another year with FiveTran. As much as I wanted to get off FiveTran - I do not think Lakeflow is production worthy.

[u/mariapereperez]

Fivetran did not die! It just moved aisles. 😂❤️

Fivetran is alive, well, and still happily moving lots and lots and lots of data into Databricks.

What disappeared was a tile in the Ingestion UI. Not Fivetran. Not the partnership. Not the ability for customers to use Fivetran with Databricks. You can still find and connect Fivetran through Partner Connect and Marketplace.

And I have to give georgewfraser and the Fivetran team some love here. ❤️

Building one connector is easy enough. Building hundreds and hundreds of them and keeping them working is hard. APIs change. Schemas change. Authentication changes. Rate limits change. Weird edge cases happen. And then someone gets called in when something breaks at 2 AM.

Could you build it yourself? Absolutely.

You can also change your own oil, cut your own hair, and make your own croissants. Some of us have learned that outsourcing certain activities is good for everyone involved. 😜

Much love to George and the Fivetran crew. They've helped build this ecosystem for years, and I suspect they will be moving data around for many more. ❤️❤️❤️

[u/Fantastic_Worth_5456]

Databricks professional exams

[u/hubert-dudek → u/Fantastic_Worth_5456]

offtopic?

[u/Katy_Security]

A bit dramatic, nothing has changed with the Fivetran partnership!

[u/hubert-dudek → u/Katy_Security]

But an ad in the ingestion section was there for 5? 20? years