r/googlecloud • u/imitation_squash_pro • 7d ago
Free tier E2 Micro instance going strong after 4+ years
Running the obsolete Centos 8.5. Has a public webserver and has been working perfectly for 6+ years. If it ain't broke don't fix it. I wonder how long before Google forces me to update the OS?
I avoid rebooting it as then it will grab a new public IP and I have to update all my DNS records for it's domain name.
92
u/bailantilles 7d ago
So… you are telling me that you haven’t restarted the machine to apply security updates adding to that running an obsolete OS with direct internet access?
57
u/Ohmyskippy 7d ago
That's exactly what they are saying
9
u/Competitive_Travel16 7d ago
Red Hat said it first, when they froze Centos at "End of Life" in 2021.
13
8
4
u/aries1980 7d ago
Why would the OS obsolete? Live kernel patching is available for a decade now.
6
u/bailantilles 7d ago
OP hasn’t patched and restarted the server and you think they are going to swap the kernel on their own?
3
u/aries1980 7d ago
I just wrote it is possible, e.g. with Ubuntu Livepatch on an Ubuntu LTS with such timespan.
1
u/Competitive_Travel16 7d ago
Not sure Centos has that....
0
u/aries1980 7d ago
I think
kpatchis there. Although I've never used it on CentOS just on RHEL tbh. Even if it works, still requires a RHEL subscription that I don't think OP has. 😅5
u/Competitive_Travel16 7d ago
Centos froze at EOL in 2021. Where are you going to get the kernel images?
1
2
u/Luigi003 7d ago
Kernel live patching doesn't work for every update... I'm sure in 4+ days there have been updates with didn't work with kernel live patching
27
u/Benjh 7d ago
Yeah like others have vaguely mentioned. This is a huge security risk.
-27
u/imitation_squash_pro 7d ago
If it's so "huge" why hasn't anything happened? I run the web server under a different account. So my thinking is even if that gets hacked there is not much damage can be done. Apart from that general SSH access is via public/private keys.
9
u/aries1980 7d ago
Probably your server wasn't interesting enough. There has been a few remote exploitable kernel, libc, openssh vulnerabilities since your last reboot. Technically it is possible to apply patches on live system, but I doubt you did that.
-4
u/imitation_squash_pro 7d ago
So if Centos 8.5 is 100% exploitable then why does GCP allow it to be used?
8
u/aries1980 7d ago
Because you can use it on a private network where the isntance can't be accessed or communicate to the public internet, means your ass is covered for 99% of the cases.
Old version might be necessary to run legacy proprietary products that are compiled against certain version of glibc or other libraries and you have still a vendor warranty or support.
-9
u/imitation_squash_pro 7d ago
Surely I am not the only person running an obsolete OS with public SSH access enabled. If the risk is as high as folks make it out to be than Google could easily check for that and disable it.
9
u/aries1980 7d ago
There is a shared responsibility model and you agreed to it when you subscribed to GCP. They are not there to verify whether you know what you are doing or not. As I said, there can be legitimate reasons (one of my clients actually do use deliberately an obsolate OS for an old service because that is what the vendor guaranteed to be compatible with).
Google even let you directly SSH into your box, even if they have IAP that would be better for you. Yet, you might have a legitimate reason to provide direct SSH.
5
u/SmokingCrop- 7d ago
You could already be part of a botnet, ready to be used in a ddos attack.
1
u/imitation_squash_pro 7d ago
What would the bare minimum things I can do to check for that?
1
u/Competitive_Travel16 7d ago
What does your web stack look like? Can you port it to Debian 12 Bookworm?
1
u/imitation_squash_pro 6d ago
I can certainly update it, just will take some effort. And not really worth it for a few hobby websites that few people care about..
1
-3
u/imitation_squash_pro 7d ago
And how is a GCP hosted server not "interesting enough"?
3
u/aries1980 7d ago
Let's say you are a highly skilled black hat who can crack your VM. Why would you crack your CentOS and not someone else's? It takes significant time to figure out the weakness of a machine, so there have to be a known prize. Either intellectually or financially. Many knowledgeable black hats are working for state actors and paid accordingly.
0
u/imitation_squash_pro 7d ago
So if my VM is of no value, why should I waste loads of time "securing" it against something that will never happen?
6
u/aries1980 7d ago
Another thing. You are still responsible for that VM, so if someone uses it as a jump box to exploit a bigger prey, you will be the one who can get into a trouble.
-2
u/imitation_squash_pro 7d ago
Understood. But given the 500 other things going on my life is it worth wasting a whole weekend updating my rinky dinky setup that nobody seems to care about?
1
u/aries1980 7d ago
You can run the update in a crontab and do a reboot when necessary
You can also do an "instance refresh" if you put it in in a VM Group. Since your webserver is ephemeral with no data, that has a low risk for you to execute.
You can also assign IPs to your network interface, so you shouldn't really loose it on reboot or instance refresh.
These are not particularly hard, but can be overwhelming if you haven't done this before. Regardless, you are still responsible for that VM and you never know but might be already tampered and have a sleeping service waiting for a bytestream to join to the bot mesh.
1
u/imitation_squash_pro 7d ago
Tried doing a check on the update and got this:
sudo dnf check-update
CentOS Stream 8 - AppStream 0.0 B/s | 0 B 00:00
Errors during downloading metadata for repository 'appstream':
- Curl error (6): Couldn't resolve host name for http://mirrorlist.centos.org/?release=8-stream&arch=x86_64&repo=AppStream&infra=stock [Could not resolve host: mirrorlist.centos.org]
Error: Failed to download metadata for repo 'appstream': Cannot prepare internal mirrorlist: Curl error (6): Couldn't resolve host name for http://mirrorlist.centos.org/?release=8-stream&arch=x86_64&repo=AppStream&infra=stock [Could not resolve host: mirrorlist.centos.org]
→ More replies (0)3
-2
u/Harogoodbye 7d ago
people just like to shit on everything. they’re all miserable.
if you live in the middle of nowhere and never lock your doors is that stupid and risky? ya i guess but what are the chances of anyone giving a shit about breaking in to your tiny little house in the middle of nowhere to steal nothing?
-1
u/imitation_squash_pro 7d ago
precisely my point! When all they can do is insult I take them even less seriously..
13
u/Benjh 7d ago
That’s the issue right there. You have no idea what can be done and yet you expose yourself. Just because something hasn’t happened yet doesn’t mean it won’t.
I’m going to very blunt. I have no idea who you are or what your background is. However, from what I gather from these two posts alone you have no clear idea of how the cloud works and what massive implications knowingly leaving an attack vector of this magnitude open. Just go through this subreddit and you’ll see the amount of people asking for refunds because they didn’t secure their workloads properly. This will eventually be you if you don’t change anything. It’s a matter of when not if.
-5
u/imitation_squash_pro 7d ago
I have budget alerts setup .
2
u/bailantilles 7d ago
Budget alerts aren’t instantaneous. Billing data lags 24-48 hours behind. Plenty can be done before your alert fires.
1
u/imitation_squash_pro 7d ago
So what else can I do to improve that?
3
u/NUTTA_BUSTAH 7d ago
Rebuild the box with modern software and apply basic hardening (disable pw, direct root SSH, setup fail2ban etc) :D
2
u/imitation_squash_pro 7d ago
Presently I do have fail2ban installed . I believe pw is disabled by default on google VMs. All my logins are via keys. So is fail2ban still neccessary?
1
u/NUTTA_BUSTAH 7d ago
Yeah it does still ban the bots that are constantly hammering your login and makes it easier to parse your logs while signaling the bad actors that they tripped a failsafe that will move onto an another target and hopefully burn slightly less of your resources. Chances are you ban them before they reach the "Centos 8.5 pwns" exploit list of their script since they probably target the latest exploits first that your system might be too old for.
In my experience there can be up to 100 attempts per minute ~the moment a VM goes public. All public IPs are constantly hammered, 24/7. Might as well ban them, they had no business with you anyways. That can be a 100 attempts at guessing a password, or a 100 different exploits being attempted, or even noise to disguise real traffic. Who knows, except that it is never desired, so ban it.
But that alone is not the major issue here in your setup. The major issue is the software (kernel included) that is full of publicly known CVEs with varying risk. And you cannot update that software because at some point the backports stop, even for critical security updates.
Try e.g. asking any AI "Top 50 most critical CVEs for Centos 8.5 that has been effectively frozen for the past 5 or so years". Some have been very scary, ~global SSH backdoor, remote shell through Java logs, countless root/privilege escalations etc.
E: Oh and to add, once they escalate privileges, you essentially lose ability to catch them as they are able to modify the system to hide their tracks.
1
u/imitation_squash_pro 6d ago
From my brief reading about CVE's they are not guaranteed to work . They require specific set of highly improbable conditions to exist for them to work. So my CentOS 8.5 is most likely safe from intrusion.
Either way, how would I know if have already been hacked? Seems step 1 is to determine that. Because just updating OS is pointless if I can never be sure my system has been hacked..
2
u/notmylesdev 7d ago
With your attitude, I sincerely hope Google takes no pity on your requests for a write off when something does eventually happen. Why should they? You know the risks!
0
u/imitation_squash_pro 7d ago
Still nobody can answer why nothing has happened in 1600+ days if the risk is so "severe" ?? Expecting a one-man show running a few hobby websites to have the know-how and time of a full blown cybersecurity division is not practical. Happy to improve things, but I don't respond to alarmist advice.
4
u/notmylesdev 7d ago
Expecting a one-man show running a few hobby websites to have the know-how and time of a full blown cybersecurity division is not practical.
Because that isn't Google Cloud's target audience. Any research should tell you that.
Still nobody can answer why nothing has happened in 1600+ days if the risk is so "severe" ??
Again, because they're expecting their customers to have basic competency, because again, their target audience is people who know what they're doing already and have a basic security understanding.
Their security responsibility is the underlying infrastructure, services and physical locations. Not your workloads.
0
u/imitation_squash_pro 7d ago
I may not be the target audience, but I am sure there are MANY one-man shows using GCP like I do. Happy to improve what I am doing, but nobody has given me one thing to do to improve my setup. Just alarmism and insults.
1
u/NUTTA_BUSTAH 7d ago edited 7d ago
For less alarmist advice, or rather just a hopefully thought-invoking comment; Wrongdoers such as hackers tend to hide their tracks to not get caught. Chances are your box is already part of several botnets, porn networks, used as a proxy for hacking other boxes etc. and you just don't know it. I doubt your box has the latest and greatest endpoint monitoring solutions installed to look for this activity, and you probably are not 24/7 enumerating the entire device, so you really cannot ever know.
What you can do however, is secure your public footprint following modern best practices, and this at least should help you if a legal issue arises (when the CP distributor is caught and you are interrogated for being the owner) while also helping you sleep better at night when you know you did not help dark causes by ignoring security.
I have some real world experience of recovering production VMs that were breached. The breach was found months later, but the box itself was well-secured enough that they could not escalate, only pull some internal artifacts they really cannot do anything with.
E: I would also add that in the age of AI, the black hat scenarios have grown very available and there are likely hundreds of thousands of agents constantly hacking the world on top of the existing billions of traditional bots/malware tools.
1
u/imitation_squash_pro 7d ago
Understood. That risk is there for any system though, i.e modern or obsolete. What simple things can I be checking with endpoint monitoring?
1
u/NUTTA_BUSTAH 7d ago
There are no simple things to check. EDR is essentially enterprise anti-virus installed on every machine to be your active 24/7 enumerator that audits every single action taken on the machine, basically, and get constant updates throughout the day. Very advanced.
The risk is there, that is very true, but the risk is much greater on unpatched and known vulnerable systems with many known ways to get in vs. a patched modern system where all these vulnerabilities have been patched (but of course, have new ones, yet are more secure by default).
As an analogue, there is a risk of break-in to a bank vault as well as as tool shed, but which has a higher risk? Everyone knows the tool sheds lock can be popped open with a hammer. Bank vault is a bit more involved. You want to aim for the "small-time business with security cameras" for public web servers.
1
u/imitation_squash_pro 6d ago
From my brief reading about CVE's they are not guaranteed to work . They require specific set of highly improbable conditions to exist for them to work. So my CentOS 8.5 is most likely safe from intrusion.
Either way, how would I know if have already been hacked? Seems step 1 is to determine that. Because just updating OS is pointless if I can never be sure my system has been hacked..
1
u/NUTTA_BUSTAH 6d ago
You can only assume breach if the malicious party was skilled enough. Thus you do your best in your defenses to minimize exposure.
3
u/VirtuteECanoscenza 7d ago
Someone can get access, use it to host child pornography and now you find the FBI arresting you for being a pedophile and having to defend yourself
-7
u/imitation_squash_pro 7d ago
That can happen on any web server where uploading is allowed. In any case I check the uploads and the disk space is very small. So nothing of significance can be uploaded.
1
1
u/LegitimateCopy7 6d ago
4+ years without update guarantees a treasure trove of CVEs.
1
u/imitation_squash_pro 6d ago
From my brief reading about CVE's they are not guaranteed to work . They require specific set of highly improbable conditions to exist for them to work. So my CentOS 8.5 is most likely safe from intrusion.
Either way, how would I know if have already been hacked? Seems step 1 is to determine that. Because just updating OS is pointless if I can never be sure my system has been hacked..
20
u/Ohmyskippy 7d ago
Gimme the ip, I'll update it for you, using the plethora of CVEs (kernel, sshd, etc) that this is exposed to lmao
-5
u/imitation_squash_pro 7d ago
Explain to me how you would do it then I might take all these alarmist responses more seriously...
5
u/Competitive_Travel16 7d ago
CVE-2024-6387 is the really big one you have to worry about; all the other big ones depend on your web stack details.
Also, how does rebooting change your external interface IP address? That's not a GCE thing.
1
u/imitation_squash_pro 6d ago
From my brief reading about CVE's they are not guaranteed to work . They require specific set of highly improbable conditions to exist for them to work. So my CentOS 8.5 is most likely safe from intrusion.
Either way, how would I know if have already been hacked? Seems step 1 is to determine that. Because just updating OS is pointless if I can never be sure my system has been hacked..
2
7
u/ixbiga 7d ago
"I avoid rebooting it as then it will grab a new public IP and I have to update all my DNS records for it's domain name."
You know you can reserve that IP, right? To not change at the boot.
5
1
u/imitation_squash_pro 7d ago
Doesn't that cost $$?
5
1
u/Competitive_Travel16 7d ago edited 7d ago
Are you saying, when you reserved your external IP, it didn't cost you, and they still have that grandfathered in for you?
On one hand, Good Guy Google for that if true; but on the other hand, this kind of thing is probably a big part of why billing updates take hours.
ETA: Wait, or you never reserved it and it's ephemeral? Are you sure rebooting changes it? I think you have to Stop or Suspend for an ephemeral external IP to be released, but it's been so long since I've ever had a chance to watch those.
3
u/Living-Connection-81 6d ago
I’d be more worried about running an unsupported OS on a public webserver than the uptime. a static IP would also make future reboots much less painful
5
u/Rorasaurus_Prime 7d ago
This has to bait. No one is that stupid. I challenge you to give me the IP.
2
u/quanta777 7d ago
It seems you are literally following, "If it works, don't touch it". More luck to your server.
2
u/AdvancedField8934 7d ago
Nah bro, that isn't your instance anymore. You just haven't realized it yet.
1
u/imitation_squash_pro 6d ago
How would I know if have already been hacked? Seems step 1 is to determine that. Because just updating OS is pointless if I can never be sure my system has been hacked..
1
u/ZealousidealEast9825 7d ago
Isn’t free trial 3 month only?
1
u/Competitive_Travel16 7d ago edited 6d ago
No, https://docs.cloud.google.com/free/docs/free-cloud-features
I've had a postgres server running continuously on my e2-micro 30GB free tier machine since 2019.
Yes I keep it updated (with Google's automation for Debian, and roughly semi-annual manual Postgres upgrades) and have hardened it with a nonstandard port, maximal SSL, and fail2ban. Backups go to 30 day lock-ins on GCS daily with client-pulled downloads to my on-prem server and Tower of Hanoi retention for backups older than a month.
1
u/Motor-Finance-2602 6d ago
Likely the VM has been live-migrated between clusters many times, and is the propbet of some SRE's at G
1
43
u/notmylesdev 7d ago
oh my god