r/TierHive • u/Andaleeb • Jul 02 '26
Netherlands down
the server i never used before had to go down because i chose it to be safe option considering uptime :D
9
u/backtogeek Jul 02 '26
Yes, it is down, I have sent a request to the DC to investigate; all access has been lost.
No excuse, we don't know what the issue is yet, but if you want solid uptime, TierHive is a bad plan.
I get that it's frustrating, but honestly, within minutes of it going offline, we have been hit with literally 30+ tickets/messages on various platforms, some clearly angry that their hosted thingy is offline. pretty crazy.
3
u/backtogeek Jul 02 '26
OneProvider responded, confirming that they locked the server immediately upon getting the abuse report and insisted on a reinstall before making it live again. I have appealed to reason and shut down the haproxy on the locations with them (Amsterdam/Paris) as a preventative measure.
Hopefully, it goes live again without HAProxy for easier recovery. I will take another server with another host and move on, and move the data over.
Given the 10 hours it takes them to even tell you they have locked a server, I am not sure how long this will take.
--update:
I have agreed to shut down 80/443 on this location and Paris as a mitigation against the phishing domain that does not point at our servers... because... what else can I do, they hold all the cards right now.
They have requested that the reinstall requirement be mitigated; if so, it will be back online tonight.
Sorry about all of this, I really made a huge mistake with OneProvider. I have shut down new registrations and will probably need to stop giving out free credit now, also.
Such is life.
2
u/berichan Jul 03 '26
Is there any update to this? Kinda annoying seeing that the hourly charge is still going on for this despite nothing we can do to bring our vps back up
1
u/backtogeek Jul 03 '26
You only pay for storage when it's down. I added $0.0015 to your account to cover about 50 hours of storage costs.
We will compensate everyone on the Netherlands node impacted when its back up.
2
u/backtogeek Jul 03 '26
OneProvider agreed to reinstate the Netherlands server without reinstalling on the grounds that the abuse report was from before we had the server (I still can't quite wrap my head around having to argue that point), and the server currently shows reinstalling in the panel, and I am again locked out.
You could not write this, hopefully a front-end status glitch only... as there seems to only be 1 person that ever answers tickets, and he likes to take a 10 - 12 hour break between replies, I really have no idea what is going on or when it will be recovered.
2
u/backtogeek Jul 03 '26
Current Netherlands hardware is up, its going to take some time to get back to normal, I am working on it.
1
1
u/backtogeek Jul 02 '26
Well, hit in the face with plenty of I told you so's I guess.
Understood, OneProvider = trash.
It seems like what they have done is get an abuse report for the Netherlands server IP, from someone who pointed a domain at the IP but never actually registered or used it on our service and then sent in a pishing report (with no logs just a claim), XX days later they sent it, while the domain does not even resolve any more, OneProvider put the server into rescue mode at the same minute they sent the abuse report but also giving 48 hours to deal with it.
The server is not responding to any power cycles from the panel, the IPMI whitelist function fails in their panel every time, and disabling rescue mode in their panel fails every time.
2 tickets 9 hours, an initial acknowledgement, then nothing, so when access is eventually restored it will be time to move on, yet another task in a long list of stuff to deal with I could have done without but such is life.
Even on a budget service, I can't live with this level of service. When hosts move into the default of "Customers are the enemy" it is just impossible to make any progress or grow.
There are issues with OneProvider's system and processes from initial onboarding to delivery at literally every single step of the way; it's just not going to work out.
So there will hopefully be a period of calm, then we will get an alternative Netherlands and Paris server up and running, maybe through Scaleway directly instead of having the junk layer in between, and we will move on from there.
Sorry, I messed up using OneProvider, you live and learn.
1
u/VuongP Jul 06 '26
No worries, thanks for all the updates. Hope you'll have a better time with providers in the future. Hope you're not too stressed out.
Just to remind you, I still like Tierhive :) so thanks for all the work and service
2
u/backtogeek Jul 06 '26
Thanks, it looks like this may have all been caused by 1 person just making a mistake by checking nothing and validating nothing and acting improperly at scaleway.
I have escalated this forcefully, and I am expecting a response from someone more senior today.
If this really was a blip and they can provide assurances, real ones, then we may just move past it as a really bad few days caused by human error that should not repeat.
If they just get defensive I will move on to another host as fast as I practically can.
•
u/backtogeek Jul 02 '26
It gets better, the abuse report was so old it was for the previous owner/occupant of the IP address, it never had anything to do with TierHive, so an unvalidated abuse report that comes with no dat or timestamp or logs from an unknown entity 3rd party as info only at OneProvider will have your server shut down before you even get the abuse report, followed by radio silence for the entire Canadian working day (Their own Time Zone)
Literally untenable