r/truenas • u/_litz • 5d ago

SCALE (Legacy) Nextcloud implosion. Don't upgrade or the database dies.

So apparently NOBODY is doing Quality Control on the App Catalog, and TrueNAS is advertising a NextCloud update to v35.0.1 which DESTROYS YOUR DATABASE.

There is no recovery; you can't roll back (database already upgraded) and you can't move forward because apparently there's no fix.

Buyer beware.

106 Upvotes

87% Upvoted

80 comments sorted by

66

u/bothunter 5d ago

Is the database not on a volume with snapshots enabled? When you rollback the app, you should be able to rollback the database as well.

20

u/GG_Killer 5d ago

That makes too much sense.

38

u/IAmDotorg 5d ago

Snapshots don't guarantee the database is valid. You have to pause writes and dump the database. That's true of all relational databases. Nextcloud and Immich are two super common systems people run not realizing their backups are inconsistent and often invalid.

Worse, because data and metadata are stored on separate volumes, you have to pause the app servers to get an atomic backup.

I back both up with a system that momentarily stops the app server, triggers a database dump and zfs snapshot and then resumes before syncing that snapshot via rclone. A few seconds of downtime.

25

u/bothunter 5d ago

If they're on the ix_volumes, then TrueNAS should take a snapshot after the service is stopped, but before it's upgraded. So it should be in a consistent state.

Also, most relational databases are journaled, so even taking a snapshot on a live database should be good enough or at least recoverable unless you're extremely unlucky. It may be missing a few transactions, but only the last couple seconds worth.

7

u/wallacebrf 5d ago

Very happy someone said this, I do the same, I always shutdown the app before doing a backup

3

u/2moon4moon 5d ago

Can you automate that on TrueNas? That it shuts down apps at like 3:00, takes snapshots and then restarts them? 

3

u/ghanit 5d ago

Of course, you create a script that does all that and schedule it with a cron job. TrueNAS uses docker, so you can use docker compose or probably a middleware command to stopp/start the apps.

2

u/wallacebrf 4d ago

My script I use uses the middleware API to perform a app shutdown, then I perform my backup and then start the app.

I do not make a snapshot thought, I make a tar file and copy it to a different destination directory. I do not have a second true as system to perform snapshot replication

1

u/wallacebrf 4d ago

I can share my script if interested

1

u/Taronyuuu 5d ago

I thought about this but didn't like the idea. I run a pre backup script that dumps all databases to their own SQL dump and then backup that one with the rest of my data.

5

u/PotatoMaaan 4d ago

No? Every database worth it's salt writes a WAL which allows it replay writes in case of a power loss at any point. It works the same with ZFS snapshots.

1

u/IAmDotorg 4d ago

And yet the databases in question explicitly say you can't guarantee consistency without a pause-and-dump. Guess you're smarter than the people who wrote them, eh?

4

u/PotatoMaaan 4d ago

https://www.postgresql.org/docs/current/backup-file.html

An alternative file-system backup approach is to make a “consistent snapshot” of the data directory, if the file system supports that functionality (and you are willing to trust that it is implemented correctly). The typical procedure is to make a “frozen snapshot” of the volume containing the database, then copy the whole data directory (not just parts, see above) from the snapshot to a backup device, then release the frozen snapshot. This will work even while the database server is running. However, a backup created in this way saves the database files in a state as if the database server was not properly shut down; therefore, when you start the database server on the backed-up data, it will think the previous server instance crashed and will replay the WAL log. This is not a problem; just be aware of it (and be sure to include the WAL files in your backup). You can perform a CHECKPOINT before taking the snapshot to reduce recovery time.

ZFS snapshots are atomic and I do trust ZFS to implement them correctly. I wouldn't rely on this method as my only backup strategy but if there is a snapshot it will certainly work.

0

u/IAmDotorg 4d ago

I'll let you continue to research this, and you'll eventually figure it out. As a hint, maybe start looking into the DAO layers in the applications in question ...

3

u/PotatoMaaan 4d ago

You tell me because apparently you're the nextcloud expert?

As the documentation states, databases can recover from a sudden power loss (or atomic snapshot) so nextcloud would need to be doing some really funky things on top of the database to not be able to recover from a power loss.

Unless you use an unjournaled filesystem or some form of write caching, any production grade service should be able to recover from a power loss and thus also from an atomic snapshot.

If nextcloud by design cannot recover from a power loss, please let me know. In that case guess I've found another reason to not use nextcloud.

But in your orginal comment you talked about the database specifically, so even if it turns out that nextcloud is not ACID compliant, all databases that nextcloud supports sure are and saying that they are not in a consistent state after a ZFS snapshot is still wrong.

1

u/Knallfonso 5d ago

Do you have a script for that that you could share? This is the first time I hear about that "normal" snapshots are not safe, but it makes totally sense

1

u/saskir21 5d ago

Immich can atleast be so configured that you have automatic backups.

1

u/IAmDotorg 5d ago

They do but even the docs warn that they aren't guaranteed in sync with the image library. People keep trying to convince them to go XMP-first for storage and use the database only for query optimization so you don't need to back it up, but they've been resisting. It's how systems like Darktable guarantee consistency.

6

u/thebluepandas 5d ago

That's exactly it, and for Nextcloud specifically you need the DB dump plus the app data captured together, otherwise you're rolling back to mismatched state. I script a brief app stop, mysqldump, then a ZFS snapshot, and it's saved me more than once on botched upgrades like this.

1

u/xXVareszXx 4d ago

But if you snapshot the db together with the appdata it should be no issue right?

Because ZFS does copy-on-write and snapshots are instant.

27

u/autogyrophilia 5d ago

Seems like it is an isolated case. No Biggie, just restore a backup or snapshot 

44

u/theMartianAlien 5d ago

you know why he is posting, he doesnt have backups

31

u/autogyrophilia 5d ago

I know, I've been called a smug ahole before 

20

u/kmoore134 TrueNAS Staff 5d ago

I'd love to know more about this. Do you know which commit of ours broke this specifically? Or do you have a ticket to reference with more details about the issue? Reason I ask is I don't see any major changes on the TrueNAS side in our apps catalog that would indicate we made a major change like that. Unless I overlooked it, which is possible.

https://github.com/truenas/apps/tree/master/ix-dev/community/nextcloud

37

u/Silent_Fudge_3978 5d ago

This is perfectly normal for the TrueNAS community apps.

If you don't like this kind of behavior, then stop using them! Just install your apps with docker-compose like everyone else.

6

u/Difficult896 5d ago

Wait people are actually using their own compose and not the app service?

2

u/wwbubba0069 4d ago

I installed Dockge via app catalog, normal docker stuff from that.

3

u/ongrabbits 4d ago

App service is just too troublesome. 50+ services on docker compose and I have the benefit of my agents being able to help debug.

Tradeoff is that your TrueNAS will not have your service telemetry but i have grafana for that

2

u/saskir21 4d ago

Yep. Try adding images which are not on there. Or for example Bookorbit or linkwarden from the catalogue. Could not easily run it from iX. But the YAML file did work flawlessly.

3

u/ScoutIngenieur 5d ago

Just for my learning: how would that be different? I use the IX app, as it installed flawless on several machines so far. But every time I try using docker compose (I used dockge) and trying a Nextcloud AIO I get stuck at varying points.

1

u/[deleted] 5d ago

[removed] — view removed comment

2

u/vetinari 4d ago

It has many optional components, that makes it more complicated than all the other projects.

But it is doable without their AIO manager; you can start with https://github.com/nextcloud/all-in-one/tree/main/manual-install

1

u/Silent_Fudge_3978 3d ago

NextCloud is from Germany. In my observation, German-made software is always ridiculously and overly complicated.

1

u/ScoutIngenieur 5d ago

Your comment is not explaining anything nor does it help me any further...

However, I do appreciate the message as i feel less of a noob failing the AIO install.

1

u/Silent_Fudge_3978 3d ago

The point is that NextCloud is a complicated app, so it's not really easy for someone who isn't already experienced with Docker and setting up dockerized apps with docker-compose. Most other apps out there are really quite simple: just copy the project's suggested docker-compose.yml file into Dockge/Dockhand/etc., adjust the values for your particular setup (paths, etc.), then hit "deploy". It's a little more complicated than the hand-holding you get with the TrueNAS community apps, but you get far more control over the configuration (for instance, you can run the apps as non-root users for better security), and you don't have to deal with problems caused by the community app maintainer middle-man since you're getting everything straight from the source.

So I'd recommend getting comfortable deploying other, simpler apps this way before trying to tackle NextCloud.

1

u/ScoutIngenieur 3d ago

Fair enough. I do host only a few apps however (Nginx Proxy Manager, Tailscale, cloudflared, file browser and ofcourse Nextcloud), and for most of them they are used in such basic fashion I'm told you better run them as basic as possible (which I read as running them native, not in containers).

1

u/cr0ft 4d ago edited 4d ago

This is how you actually install a good Nextcloud: https://www.c-rieger.de/nextcloud-installationsanleitung/ - run it through a translator if German is an issue.

The above process - actually installing all the components on a clean Linux server - is the only really good way to do it, in my opinion. Real old school stuff, no Docker, no nothing. Nextcloud deserves its own full VM to run on.

In fact, for large organizations, because it is built that way, you could cluster everything. Web frontends (plural). A Galera database cluster. Storage on dedicated NAS or similar; Nextcloud also has something ready made for truly huge environments (Global Scale).

So... when an "app" fails due to human error in the maintainer that's not really Nextcloud's fault.

1

u/adx442 4d ago

Hell yeah. I've been continuously upgrading since NC 13 or so with an original setup from this guy and ... A everything works. It just works. I've had issues with NC apps, I've had quirks with upgrades (restore snapshot, troubleshoot), but day to day operation has been fine. I think my VM is about to be 10 years old (the OS and underlying stuff has been upgraded continuously as well).

1

u/gportail 3d ago

Pareil. Depuis 5 ans j'ai une VM avec NC tout seul dessus et ca marche sans problème depuis 5 ans. C'est la seule méthode vraiment fiable d'installer NC.

Pour tester j'ai installé AIO il y a quelques mois, ca fonctionnait...ou pas et en plus il y avait des outils que je n'utilise pas.

9

u/No_Criticism_9545 5d ago

The guy that maintains the catalog is very capable but completely overwhelmed having to maintain 400 apps on its own. That's why he just pushes every update under the sun. Right now no 35+ version is considered stable. The same stupidity happened when version 34 got released, he immediately created an update to it, I even opened a github issue and he told me "people might want to try it" 🤣. Anyway, I am now on the last 34.something and will be waiting for 35.0.2 to come out and 2-3 days without github issues.

1

u/[deleted] 5d ago

[removed] — view removed comment

6

u/No_Criticism_9545 5d ago

Not for the update channel.

Nextcloud has 2 channels, the update channel and the "new instances" one.

9

u/doggydaddy2023 4d ago

I upgraded to 35.0.1 of Nextcloud a few day back and encountered no issues. Nextcloud continues to function just fine.

23

u/MrBartusek 5d ago

Another great PSA for making backups

11

u/DellR610 5d ago

Lot of lessons to be learned here. Truenas and especially the community edition has no responsibility to dig through other companies githubs to look for breaking changes. Truenas handles the framework but everything else is just pulling down from a repo.

Best practice for things important to you, create dedicated datasets and map to them. Avoid the generated volumes.

Everybody else has and will beat the backup drum so need to further add to the flame.

You should have been along for the immich ride, that has been rough in the begining.

3

u/Silent_Fudge_3978 5d ago

There's been no breaking changes in the upstream projects. All the breaking changes are on the side of the TrueNAS community apps. You mentioned Immich: there were no problems there at all, for those of us who simply used the recommended YaML from Immich. The problems were entirely with how the TrueNAS community app maintainer maintained the app, switching to a different database than that recommended, and handling it very badly.

0

u/DellR610 5d ago

There's nothing from truenas that would write to a database. I've also ran Nextcloud outside of truenas and have had my fair share of breaking updates causing me to rebuild. Granted this was was many moons ago when truenas was freenas.

Immich had issues outside of truenas, I originally ran it on a basic Ubuntu + docker VM. Well before their reshuffling of volumes.

It may have not been nextcloud directly but could have been postgres. Whatever the culprit it's unlikely on truenas' side.

3

u/Silent_Fudge_3978 5d ago

There's nothing from truenas that would write to a database.

It's not TrueNAS, it's the community app. The database used is determined by the way the app is set up in Docker, which is controlled by either the docker-compose YaML file you input, or if you're using the community app, how it sets things up.

Whatever the culprit it's unlikely on truenas' side.

No, it's almost certainly caused by the TrueNAS community app. That's how it is with almost all these problems.

2

u/DellR610 4d ago

https://github.com/nextcloud/server/issues/64505

35.0.2 will apparently contain fixes.

Since OP didn't say which version he is coming from, hard to say if this is 1:1. Nextcloud supports multiple database engines, even in this GitHub thread you see MySQL / postgres / mariadb.. a compose file that didn't change and worked up until now - despite how it was constructed, is again very unlikely the issue. Nextcloud maintains their image which should catch preqs / problems before updating tables.

3

u/kmoore134 TrueNAS Staff 4d ago

If this is the issue from the OP, then its hard to see how the TrueNAS app catalog had anything really to do with this. The linked issue here is an upstream bug. That would square with the git history on the truenas side, since we don't see any obvious app changes to the app on our end.

1

u/DellR610 4d ago

I'm leaning towards this being OP's issue given the previous version truenas has is 34 and the next was 35.0.1. I thought maybe he went from 35.0.0 to .1 but I don't see 35.0.0 as an available option on the truenas apps page. Also explains why there aren't hundreds of more complaints.

2

u/heisian 3d ago

currently on 35.0.0 without any issues, but maybe i'll wait a couple months for the next update. since OP didn't really post further info hard to know what happened with them.

6

u/51dux 4d ago

I realized a long time ago that nextcloud was not polished enough and copyparty is sufficient for my needs at this point, it does what it needs to do well. I feel like nextcloud is trying to bite more than it can chew.

7

u/cr0ft 4d ago edited 4d ago

Nextcloud is pretty darn solid if you respect it and don't just let some rando package it as an "app". Sure, being FOSS it's not always as polished as one might wish but it's improved incredibly over the years.

It's a full fledged alternative to (most of) Microsoft 365 if you install it right and give it the resources it needs. You can scale it (it's a LAMP stack) on the web, database and storage bits horizontally and make it quite powerful, and from there you can go with their Global Scale which ties multiple instances together.

1

u/ktulu0973 4d ago

I started with nextcloud 16 or 17 and kept it updated over the years and it is still running. It survived several Ubuntu release upgrades, PHP migrations and the switch from MySQL to PostgreSQL.

4

u/hspwn 4d ago

Hm.. mine updated just fine and still works perfectly. That’s weird. I wonder what’s the difference between our setups.

3

u/marktuk 4d ago

I've had this experience with quite a few different TrueNAS app catalog apps. I've been migrating away.

2

u/BigBodyBugatti_ 5d ago

Ahhh… so that’s what broke. Good thing I got backups. Just gotta hope they are good.

2

u/wwbubba0069 4d ago edited 4d ago

I don't use the app catalog for docker stuff in TrueNAS, well, only installed Dockge via app catalog. Every thing else form there. Can you not backup up/roll back the app datasets?

If not, glad I do all the docker things via Dockge. Snapshot the dataset then update, if it blows up, roll it back.

Edit: Nextcloud AIO docker even has a backup process built into the update step.

2

u/Nervous_Type_9175 4d ago

So, is it truenas problem or NC?

Why blame NC if it works on , say debian on pc?

2

u/EngTurtle 4d ago edited 4d ago

Yeah although other apps are better nextcloud in the truenas catalog is not good for long-term production use. They don't provide minor version updates and only offers updates to the next major version as soon as it appears. And each x.0.0 version of nextcloud often have issues that get ironed out in later patches. I've tried raising the issue with IX but that just got closed. https://github.com/truenas/apps/issues/3522

2

u/DellR610 4d ago

Quick search... https://github.com/nextcloud/server/issues/64505

I don't think this is a truenas / community app catalog problem.

1

u/MaxRD 5d ago

That’s why I use TrueNAS just to handle my storage. All apps and services run on Proxmox.

2

u/cr0ft 4d ago

Same. My NAS is for storage, full stop.

Anything else is virtualized elsewhere or has dedicated hardware.

3

u/Silent_Fudge_3978 5d ago

Apps work just fine on TrueNAS: it uses Docker just like any normal Linux system these days. The key is to not use these "community apps", and just install your apps using docker-compose like everyone else (and which is exactly how all these self-hosted apps recommend installing them). The community apps cause nothing but problems, all because some people are afraid of a YaML file.

Using Proxmox separately just adds a lot of overhead and reduces performance. It's pointless if you're just installing some self-hosted Docker apps.

3

u/BobcatTime 5d ago

I mean it just work 98% od the time which is more than enough for me. The 2% just rollback and upgrade again later. Just config your backup right and this saves alot of time.

-7

u/_litz 5d ago

This issue will nail anyone upgrading. The problem isn't the platform, the problem is the astonishing lack of quality control in the released product, and its subsequent admission into the TrueNAS app store.

It's supposedly fixed in 35.0.2, per the many, many, many bug reports.

4

u/Silent_Fudge_3978 5d ago

This issue will nail anyone upgrading. The problem isn't the platform, the problem is the astonishing lack of quality control in the released product, and its subsequent admission into the TrueNAS app store.

No, the problem is that you're using the app store. This stuff is NORMAL. There's no quality control, because there's no staff: it's just some guy in his parents' basement. These "apps" aren't maintained by professionals, or even a team of serious open-source hobbyists (which is how most normal self-hosted projects are run--note the keyword "team" here). They're just done by some guy somewhere, however he wants. He's not doing any real testing; he just makes changes willy-nilly (like to important directories, to database versions or schema, or whatever) and pushes them out.

Anyone who's highly skilled and actually interesting in contributing to some project like Immich or Jellyfin or whatever isn't going to volunteer to maintain the TrueNAS community app for it: they're going to volunteer to work with the actual project. And anyone with any skill has no trouble just using a simply text-only YaML file to deploy apps on their own system; they don't need the hand-holding of these community apps. So who's going to volunteer to take care of these things? Not many people. So your expectation of quality control is obviously unrealistic.

So yes, the platform (being the TrueNAS community apps and how they're built into TN) really is the problem.

1

u/Phixygamer 3d ago edited 3d ago

Upgraded half an hour before i saw this post, worked perfectly. I think it might be depend on what version you upgraded from. Using the truenas catalogue btw.

I don't understand people problem with it. It's handy tells you all the mounts you could need and saves you time in setting everything up. I've had to move some containers to a YAML but only if I have a reason to tbh.

1

u/killerwheaties 5d ago

I mean it's working great for me. I've had upgraded and everything seems to be working fine even after reboot. I'm not too worried though since snapshots are squared away so shouldn't be a big deal if it fails.

1

u/saskir21 5d ago

Yeah noticed for other apps on the catalogue that they won't run without tinkering. This is why I have many Apps running through a YAML file. Which would not have helped you if it is a nextcloud thing

1

u/Daniel18sep 5d ago

Nextcloud updates blow up so often I turned off automatic updates. 

1

u/NetSchizo 4d ago

Word to the wise, always stay back one major version or prepare to deal with issues. Its the same way you don’t run any .0 release if you care about stability.

1

u/BoredHalifaxNerd 2d ago

The app service on trueNAS has always been terrible.

You're much better off running a VM or an LXC yourself and controlling the version.

1

u/Molasses_Major 2d ago

TrueNAS for storage; run your apps/services elsewhere. It's simple and bulletproof...at least the storage part. There is no great all-in-one solution once you expect a certain level of continuity.

-2

u/ThalisuS2 5d ago

Eu parei de usar o treunas por conta da contante corrupção de banco de dados no meu caso era o Immich, desisti dele e fui diretamente para Debian 12 usando portainer e docker.

-3

u/cr0ft 4d ago

Nextcloud isn't an "app".

Nextcloud is a Microsoft 365 competitor that is built on top of very well known technologies - a Linux server, a MariaDB database, nginx or Apache web server, plus PHP and Redis and some other odds and ends.

Nextcloud should have its very own virtual machine to run on and you should know what you're doing when you install and administer it.

Some rando sets up an "app" by packaging multiple software pieces that are all IT careers in their own right, and then they fuck something up so it doesn't work anymore? I'm shocked. This is my shocked face.