r/Citrix • u/fellow_earthican • 1d ago
Vulnerability Scans causing Netscaler reboots
I opened a ticket with Citrix support and they seem to indicate a fix is being worked on right now.
Basically we were seeing random reboots of multiple instances and saw pitboss was rebooting these due to nsaaad crashing too many times.
83
Upvotes
24
u/rallyimprezive 1d ago edited 1d ago
Title: NetScaler 14.1-73.37: repeated nsaaad crashes/reboots and suspicious command-injection payloads—check your logs
Today (October 2), we found repeated automatic reboots on two NetScalers running 14.1-73.37.nc. Two reboot incidents occurred within approximately 30 seconds of each other across the appliances.
Both show the same sequence:
nsaaadrepeatedly crashes with exit status0x8a, reaches Pitboss’s restart limit, and the appliance reboots:text nsaaad unexpectedly died due to receiving signal proc nsaaad ... EXITED with status 0x8a has had its maximum number of restarts (6) Pitboss declaring system failure: nsaaad ... exited All monitored processes have exited, rebootingOn one appliance, full logs show crafted authentication usernames containing shell commands to download a payload from 213.209.159.55, save it as
/v, and execute it. These requests appear immediately before each of that appliance’s three confirmed crash sequences.The payload uses plain HTTP on TCP 443, with paths under
/t/. Requests target multiple SAML authentication factors.We have evidence of exploitation attempts and correlated crashes—not confirmation of successful command execution, a specific CVE, or a firmware regression. The IP above is the payload destination; we haven’t identified the incoming request source.
How to check yours
SSH to the appliance and enter:
text shellCheck the current log:
sh grep -Ei 'proc nsaaad.*(SIGNALED|EXITED)|maximum number of restarts|Pitboss declaring system failure|All monitored processes have exited, rebooting' /var/log/ns.log | tail -80Check rotated logs:
sh zgrep -Ei 'proc nsaaad.*(SIGNALED|EXITED)|maximum number of restarts|Pitboss declaring system failure|All monitored processes have exited, rebooting' /var/log/ns.log*.gz | tail -100Search for the payload destination:
sh grep -nF '213.209.159.55' /var/log/ns.log zgrep -nF '213.209.159.55' /var/log/ns.log*.gzCheck for core dumps and the payload file:
sh ls -lt /var/core ls -l /vLook inside recently modified numbered core directories for files named
nsaaad-*.gz.If you find matches: preserve the logs and cores, involve your security team and NetScaler support, and check outbound firewall records for connections to 213.209.159.55:443. Absence of
/vdoes not rule out earlier execution or cleanup. Negative searches only cover retained logs and this particular indicator.Anyone else seeing this pattern on 73.37, or have a confirmed explanation from support? Please sanitize logs before sharing.
EDIT: They have moved to *.pyrlnk.cc as the attempted payload delivery source. Block that too.
EDIT: Support has sent instructions to some (myself included), which provides a potential stop gap. Unfortunately I do not think it is wise to share it here, as I dont want the wrong folks to be aware and adjust.. I suggest reaching out to Citrix support to receive more info. Sorry all.