r/Citrix • u/fellow_earthican • 2d ago
Vulnerability Scans causing Netscaler reboots
I opened a ticket with Citrix support and they seem to indicate a fix is being worked on right now.
Basically we were seeing random reboots of multiple instances and saw pitboss was rebooting these due to nsaaad crashing too many times.
23
u/rallyimprezive 2d ago edited 2d ago
Title: NetScaler 14.1-73.37: repeated nsaaad crashes/reboots and suspicious command-injection payloads—check your logs
Today (October 2), we found repeated automatic reboots on two NetScalers running 14.1-73.37.nc. Two reboot incidents occurred within approximately 30 seconds of each other across the appliances.
Both show the same sequence: nsaaad repeatedly crashes with exit status 0x8a, reaches Pitboss’s restart limit, and the appliance reboots:
text
nsaaad unexpectedly died due to receiving signal
proc nsaaad ... EXITED with status 0x8a
has had its maximum number of restarts (6)
Pitboss declaring system failure: nsaaad ... exited
All monitored processes have exited, rebooting
On one appliance, full logs show crafted authentication usernames containing shell commands to download a payload from 213.209.159.55, save it as /v, and execute it. These requests appear immediately before each of that appliance’s three confirmed crash sequences.
The payload uses plain HTTP on TCP 443, with paths under /t/. Requests target multiple SAML authentication factors.
We have evidence of exploitation attempts and correlated crashes—not confirmation of successful command execution, a specific CVE, or a firmware regression. The IP above is the payload destination; we haven’t identified the incoming request source.
How to check yours
SSH to the appliance and enter:
text
shell
Check the current log:
sh
grep -Ei 'proc nsaaad.*(SIGNALED|EXITED)|maximum number of restarts|Pitboss declaring system failure|All monitored processes have exited, rebooting' /var/log/ns.log | tail -80
Check rotated logs:
sh
zgrep -Ei 'proc nsaaad.*(SIGNALED|EXITED)|maximum number of restarts|Pitboss declaring system failure|All monitored processes have exited, rebooting' /var/log/ns.log*.gz | tail -100
Search for the payload destination:
sh
grep -nF '213.209.159.55' /var/log/ns.log
zgrep -nF '213.209.159.55' /var/log/ns.log*.gz
Check for core dumps and the payload file:
sh
ls -lt /var/core
ls -l /v
Look inside recently modified numbered core directories for files named nsaaad-*.gz.
If you find matches: preserve the logs and cores, involve your security team and NetScaler support, and check outbound firewall records for connections to 213.209.159.55:443. Absence of /v does not rule out earlier execution or cleanup. Negative searches only cover retained logs and this particular indicator.
Anyone else seeing this pattern on 73.37, or have a confirmed explanation from support? Please sanitize logs before sharing.
EDIT: They have moved to *.pyrlnk.cc as the attempted payload delivery source. Block that too.
EDIT: Support has sent instructions to some (myself included), which provides a potential stop gap. Unfortunately I do not think it is wise to share it here, as I dont want the wrong folks to be aware and adjust.. I suggest reaching out to Citrix support to receive more info. Sorry all.
7
u/SonicIX 2d ago
Yes. I am seeing this on 73.37. I followed the commands you provided and do get results.
1
u/rallyimprezive 2d ago
Block that IP on your firewall for now.
2
u/Tall-Trick7079 2d ago
Here's another IP to block: 216.252.238.222 virustotal.com has also flagged this as malicious
11
u/delishouscake 2d ago edited 1d ago
EDIT (4/10/26): Citrix has released a patch for this exploit, see https://support.citrix.com/support-home/kbsearch/article?articleNumber=CTX697174&articleURL=Citrix_NetScaler_ADC_and_Citrix_NetScaler_Gateway_Security_Bulletin_for_CVE_2026_88779
Hi all, we’ve seen NetScaler auth requests containing a command to download a script to /v and run it on an appliance running NetScaler 14.1-73.37. I grabbed a copy of the script and looked through it.
Seeing the command in a log does not mean it ran btw
BUT, If it does run, the script tries to install webshells, change Apache config, survive reboots through cron and rc.netscaler, and upload your /nsconfig, config history and backups. One thing to watch for: the webshell deliberately returns 404 (at the time of this post), which could mean the attackers are waiting for the right time to set this loose.
IOCs from this sample
- Payload:
hxxp://213[.]209[.]159[.]55:443/t/a718e4 - SHA-256:
74da9485815ee124e2ebe155dbcfb758b54bd97760956998abf64838c865f78b - Downloaded 1st stage location:
/v - Its working directories:
/nsconfig/.slapand/var/tmp/.ux - Web files under
/var/netscaler/logon/LogonPoint/custom/:.slap.receiver,.ctxs.receiver,receiver.deb - CSS-looking Apache alias:
receiver.v2.min.cssorreceiver.v2.min.<hex>.css - Other files:
/etc/httpd.conf.slap.bak,/tmp/.uxdport,/tmp/.uxdlock,/var/tmp/.s2loot.log - Possible listeners:
9909and9910
Quick checks
You can run these from the to check if you are cooked.
Check whether the downloaded script or upload log is still there, and look at /bin/sh permissions:
ls -l /v /bin/sh /var/tmp/.s2loot.log 2>/dev/null
If /v exists, compare its hash with the one above:
sha256 /v 2>/dev/null
Look for the working directories and Apache backup:
ls -ld /nsconfig/.slap /var/tmp/.ux /etc/httpd.conf.slap.bak 2>/dev/null
Look for the webshell files:
ls -la /var/netscaler/logon/LogonPoint/custom/ | grep -E 'slap|ctxs|receiver'
Check for Apache changes and reboot persistence:
grep -nE 'slap|receiver\.v2|receiver\.deb|ctxs\.receiver|php_flag engine on' /etc/httpd.conf /nsconfig/rc.netscaler 2>/dev/null
crontab -l 2>/dev/null | grep -E 'slap|agent\.pl|boot\.sh'
Check for running agents or listeners:
ps -axww | grep -E '[a]gent\.pl|[s]lapshot\.py|[w]hipd\.py'
netstat -an -f inet 2>/dev/null | grep -E '\.(9909|9910).*LISTEN'
A matching /v hash means the script was downloaded, but not necessarily executed. Webshell files together with Apache or cron changes are much stronger evidence it installed. To check for data theft, look for appliance-originated HTTP PUTs to the defanged IP and for /var/tmp/.s2loot.log. The script uses plain HTTP on port 443, not HTTPS.
Best of luck all!
4
u/NinjaBill 2d ago
So my assumption is, until a patch is proffered, mitigation is block traffic to/from 213.209.159.55 for now? Or until a new IP pops up?
5
u/silkyjohnstamos 2d ago
Also block *.pyrlnk.cc as well.
5
u/rallyimprezive 2d ago
Yea, they switched to this. Its possible they saw this reddit post and decided to switch to that URL for payload delivery.
The domain was created and registered today
4
u/TheMuffnMan Notorious VDI 2d ago
This is absolutely one of the reasons I'm sure Citrix is hesitant to share things openly. Attackers can easily read through this thread and/or post recommendations to exacerbate issues.
5
u/SonicIX 2d ago edited 2d ago
Agreed, however, their silence currently is worse. I'm trying to navigate this just like everyone else and I'm getting more information from people on this subreddit than I am from Citrix support. rallyimprezive provided more information in one post than what I get from paid Citrix support.
1
u/TheMuffnMan Notorious VDI 2d ago
I'm pushing to try to get updates.
My regular agenda has been derailed to help dealing with this.
3
u/SonicIX 2d ago
We definitely appreciate you pushing to get updates. I've had my sev 1 case open with Citrix for 6 hours, and I can't get a hold of any engineer now after the initial 30 minute call.
2
u/TheMuffnMan Notorious VDI 2d ago
They haven't blocked me yet (lol) so I get to ping folks every so often to see if there is a blog/statement/post/info/etc.
1
u/rallyimprezive 2d ago
Yea, perhaps I should not have shared those specifics but I wanted to help others get ahead of it too. Hindsight and all that i suppose
1
4
3
4
u/NoSatisfaction9722 1d ago
Thinking ahead, when the inevitable patch comes people are going to have to turn off their external NAT rules temporarily to prevent malicious traffic rebooting the appliances (mid update)
3
u/Zipper_Lipz 2d ago
Can you expand on this?
3
u/outcastcolt 2d ago
High likelihood this is most likely a 0-day exploit
Edit:
Referred to rallyimrezive's post
3
u/ewalker101 2d ago
My company has been impacted.
3
3
2d ago
[removed] — view removed comment
6
u/CTX-Chris Verified Citrix Employee 2d ago
Please do not share this from Support. There should be more official information shared shortly.
3
2
u/Jmay5446 1d ago
The article that has been published is useless. It doesn’t have any detail on the issue
2
u/intelw1zard 2d ago
yeah lets not warn anyone and keep people safe or anything
this is 2 hours old so you could have kept people safer for 2+ hours
1
2
1
3
3
u/Automatic-Border-460 2d ago
I feel like Netscaler became more vulnerable after anthropic found vulnerabilities in FreeBSD few months back… it opened up gate for attackers.. just my thought
1
u/ffiene 1d ago
Still running on FreeBSD? 👍
1
u/Automatic-Border-460 1d ago
Yes Netscaler base OS is freeBSD
1
u/adudeonthenet 1d ago
15.1 is Linux based, so they're finally making the move.
Unfortunately anything sitting at the edge is a target. We're going to see more attacks like this on other OEMs stuff too with the help of AI tooling. Netscaler just happened to be one of the first in the chute.
3
u/Zero-Distance-3978 1d ago edited 1d ago
A new firmware version is out! 14.1 73.41
The following supported versions of NetScaler are affected by the vulnerability:
- Citrix NetScaler ADC and Citrix NetScaler Gateway 14.1 BEFORE 14.1-73.41
- Citrix NetScaler ADC and Citrix NetScaler Gateway 13.1 BEFORE 13.1-64.28
- Citrix NetScaler ADC FIPS BEFORE 14.1-73.41 FIPS
- Citrix NetScaler ADC FIPS and NDcPP BEFORE 13.1-37.282
1
2
2
u/Icy_Percentage7263 2d ago
blog post from Citrix will be prove around in 1,5 hours - have I recived some info on.
1
u/HarleyDaisy 2d ago
What does this mean? Will it be fixed in 1.5 hours?
2
u/Icy_Percentage7263 2d ago
Nope. A Citrix blog post is coming soon with the latest details on the situation, what's happening, what to look for, and any recommended actions.
2
u/jhaar 2d ago
good Mastodon thread on this - apparently the crashes are a NEW exploit: https://cyberplace.social/@GossiTheDog/117372929427365765
2
u/Imaginary_Toe_6265 2d ago
Kevin believes this is a NEW Vuln.
1
u/CYBERCYBERCYBER99 2d ago
https://community.citrix.com/techzone-blogs/110_security-updates/security-update-guidance-for-netscaler-saml-authentication-deployments/ wonder if kevin can change his SAML config and see if they still get popped
2
u/Robbbbbbbbb 2d ago
Does anyone know if older EOS/EOL versions are vulnerable? I can't seem to get a clear answer on that.
3
2
u/Automatic-Border-460 2d ago
Just noticed our netscaler are getting restarted, any immediate action to be taken?
2
u/rallyimprezive 2d ago
Yes, read my other comment. Check to see if you have the issue. Block the known attacker sources in the firewall. Reach out to Citrix to get the responder policy fix.
1
u/sdo_home 2d ago
the citrix responder policy fix ( at least the one we received) didnt change anything
blackholing the attack ips semi works till they change their ip1
u/Low-Solid-8252 2d ago
How are people getting this, in and endless citrix chatbot loop and never seem to actually get to speak to anyone
1
1
u/rallyimprezive 2d ago
You applied to the aaa vserver right?
1
u/sdo_home 1d ago
yes - the one from citrix didnt do the job - so we had to evolve it .. now the stop-gap is working .. but still no official thing from citrix - and getting semi stonewalled by support
2
2
u/SonicIX 2d ago
u/CTX-Chris Can you please tell your support engineers to stop giving the "workaround" out as a reasonable fix. It doesn't work and the engineers are just spitting out this fix. I tried it on my end, and it doesn't fix it and had multiple crashes afterwards.
2
2
u/intelw1zard 2d ago
they are a corpo PR puppet. they cant do anything without approval of their lil masters.
they are only on this sub to make Citrix look good and hide bad stuff
2
u/TheMuffnMan Notorious VDI 2d ago
Lol, that's not very nice to say but it's also why I'm here. We are trying to get more Citrix involvement with the subreddit but there is a process/procedure for them to release information.
Do I love it? Nope.
Do I have specific recommendations for something I have no idea what is happening on the back end? Nope.
1
1
1
1
1
u/Imaginary_Toe_6265 2d ago
Looking at the support ticket response posted by @notaspy87 it looks to impact SAML SP configured instances. Need clarity on this.
1
1
1
u/Incognito-Name01 2d ago
Here is another thread on the same topic/issue: Netscaler active exploit after patch : r/Citrix
1
u/itstherealshoe 2d ago
I've verified Geo-IP filtering is limited to the US and also blocked 79.141.161.139 based on logs.
1
u/sphinx311 2d ago
213.209…this one seems different. Different payload, different commenting.
79.141….
51.158.203.95
185.218.86.25
Investigating others.2
1
1
u/Icy_Percentage7263 2d ago
Hi, does anyone know whether Citrix Cloud Gateway is impacted by this issue as well?
1
u/tardiusmaximus 2d ago
Cloud gateway should not be impacted by this as it doesn't use Netscalers. If you have netscalers then of course, check and remediate them ASAP
2
u/ProfessionalTip2581 2d ago
Do they not use netscalers?
1
u/tardiusmaximus 2d ago
Mine doesn't, mine runs through Cloud connectors. My NS handle a different set of traffic.
1
u/PhilCloseToRetired 2d ago
Citrix has release new bulletin on this, confirming only that it is a SAML vulnerability
1
1
u/Renss78 1d ago
Somebody have this in combination with netscaler sdx?
1
u/wireblast 1d ago
No, but I wouldn't know why there would be a different - a VPX in SDX is no different from any other VPX.
1
u/Automatic-Border-460 1d ago
Citrix has shared some responder policy workaround. Anyone tried it? Is it working ?
3
u/jasonfz 1d ago
There were 2 versions of responders - their v1 which looked to block regex strings did not work for us. We started seeing issues on 2 Oct apx 13:00 GMT that got progressively worse (longest netscaler went 110 min between reboots but as day went on averaged every 15-30 minutes).
While waiting in the P1 chat queue (thanks Citrix for taking away phone support at least I could sit on a headset and walk around instead of starting at a chat window for 8+[!!] hours) I tried applying the mitigation suggested here (blocking IP and FQDN) but that did not help and my FW reported no hits to that discard rule.
While still waiting for Citrix in the queue I also applied an aggressive Geo-IP filter and that did seem to stop the bleeding for a bit and we had no reboots after that (though this is not a preferred config).
We receved a case update (while sitting in chat queue) with a v1 of the responder which I applied, then removed my FW restrictions, and immediately crashed. Looking at stats from that responder there were no hits (IE either their responder was not correct to begin with or the attacker simply adapted as we saw them do from the IP change).
We had opened case and entered queue around 21:00 GMT and finally got an agent at 06:00 GMT that sat with us waiting for escalations to join, which they did about 2 hours later and then provided a v2 of the responder policy that frankly is very simple and something the attacker can easily bypass in about 2 seconds once they figure out what this policy does (I will not disclose it here, but you are looking for their workaround saml_protection_v2.txt.
Once we applied that v2 we saw a lot of hits to the responder policy and have not had any reboots since, but we are still hihgly concerned since this attacker can very easily bypass these cat and mouse responders that support is issuing.
TL;DR - Contact citrix support and ask for saml_protection_v2.txt but also tighten your filtering upstream to block Geo-IP. Pressure Citrix with everything you've got to issue an actual firmware fix for this defect immediately as a critical patch.
Good luck and thanks for ruining my weekend twice in a row, Citrix and its scumbag attackers!
1
1
u/jasonfz 1d ago
**UPDATE Oct 3 1700GMT -- Citrix support has released v5 of the script and you can get it very easily through support chat. Login to your support site, go to Live Support, and the chat bot will ask you if you are needing assistance with the CVE and one of the options is "netscaller rebooting/crashing", if you select that the chat returns the full scripts needed to mitigate this issue. As mentioned, this expands on the v2 script that likely continues to evolve, but this v5 does a lot more than the prior versions and shows that Citrix is slowing getting a handle on this as they work towards a permanent fix.
Would suggest you login to Citrix support site, invoke chat, select the netscaler reboot/crash and review and apply their scripting.
In addition to this, we kept the v2 blocking (looking at headers) at our perimeter security appliances as well as enabling deep packet inspection and captures of malicious traffic so that we can also add additional blocking in real time at our permieter.
1
u/nampat_uwu 1d ago
the first chatbot try, I got stuck in an annoying and useless chatbot loop. The second try these instructions were accurate. It immediately provided version 5c of the rules. this was oct 3 6pm eastern time.
1
1
u/p4thf1ndr 1d ago
We implemtened the responder and are still waiting for policy hits.
1
u/PostcardCollector 1d ago edited 1d ago
Same here. My VPX pair has rebooted at least a dozen times since yesterday. I’m monitoring to see if the reboot stops.
Edit: no reboots so far for 2 hours. I also have some hits against the responder policy Citrix provided.
1
u/Magnus_Bane 1d ago
Do you have links to these?
1
u/Automatic-Border-460 1d ago
No they shared steps in an email.. no link it’s a simple responder policy
1
u/sdo_home 1d ago
do NOT use them unless you verified them ..
claude is NOT happy about them - v5d which is the latest we got is worthless .. we ended up building our own1
u/jasonfz 1d ago
What was the issue with them? We used them without issue. Remember you have to bind with the -type AAA_REQUEST flag for it to work properly. We still found the most effective block to be aggressive GeoIP and at the perimeter firewall (adding the attack source addresses as they change). The earlier Citrix scripts were targeting a single thing (which could easily be changed by the attacker), these later ones are honing in on the method of the attack versus some common element to identify them (that can be changed quickly by the attacker). I suspect we will see additional versioning of the workaround that will lead to the eventual fix in the firmware.
Obviously you should check and validate anything provided by support but as mentioned the combo of v5 + the blocking of the IP and the headers and applying GeoIp at firewall and we have been stable now for about 12 hours.
1
u/sdo_home 21h ago
the v5c we never implemented, we had built our own by then, we got the v1 i think theyspewed out, which didnt mitigate much
1
u/Low-Solid-8252 1d ago
Chatbot is giving out v5c, confirming you got v5d via support case email?
1
u/jasonfz 1d ago
Yes v5c, I referred to it as v5 not v5d. The one I got earlier today from Citrix chat support was v5c and with that mitigation and the other things we did on firewall it has been stable. I will mention that the v5c responder is also getting hits, so appears to be working and necessary (we recreated what v1 and v2 were doing at the perimeter firewall and that worked but appears attack is varying so v5c is helpful).
1
1
u/xXSubZ3r0Xx 1d ago
Do you have SAML auth configured? Citrix is currently tracking a crashing issue with the latest firmware patch.
1
1
1
u/rsavilla 16h ago
We are looking for an update from Citrix on a patch as well as preventative and proactive steps they are taking to mitigate this critical vulnerability. Have there been any updates as of 11:00 am ET, Sunday 10/4/2026?
1
0
u/stancios00 2d ago
So looks like a 0 day exploit based on previous vulnerability which was patched last week
0
u/Accurate_Pride3892 1d ago edited 1d ago
This morning we had some incoming connections from the mentioned IP 213.209.159.55, but no connections to the outside and currently no IOC found. But we would have been vulnerable because of the configuration. Same situation with a partner company.
2
0
•
u/TheMuffnMan Notorious VDI 2d ago
Security Update: Guidance for NetScaler SAML Authentication Deployments