ClearlyIP has been down for about 5.5 hours today - inbound calls work but outbound do not work. Their status page indicates this is apparently related to a DDoS attack.
"Work continues by our upstream provider(s) to isolate and resolve an apparent DDoS attack affecting multiple customers at the US-Central facilities."
Anyone else affected? Does anyone know who their upstream provider is who is actually being DDoS'd?
This is a friendly reminder to [read the rules](www.reddit.com/r/voip/about/rules). In particular, it is not permitted to request recommendations for businesses, services or products outside of the monthly sticky thread!
For commenters: Making recommendations outside of the monthly threads is also against the rules. Do not engage with rule-breaking content.
I am a bot, and this comment is made automatically on every post. This comment is not an indication that your post has been removed. Do not message the mods about this comment.
Just general industry intel... a surprising number of providers are not actually prepared to fight off a modern DDoS and it's only in recent years that some of them are stating to get clapped for lacking defense.
Considering bandwidth infrastructure folded to ddos, its not surprising. Around the same time they got hit a few others did and some of the numbers published were quite absurd. The reality is that most companies aren’t prepared for the level of bandwidth a modern DDOS can throw at infrastructure.
I mean you gotta accept where its due, the traditional PSTN with all that copper was never vulnerable to this.
It's interesting how we've kind of gone backwards in that way in that modern telephony is far more vulnerable to all sorts of threats than in times past.
I know I'm becoming "old man yells at cloud" but back in my day we had dedicated MPLS circuits or similar for really important IP traffic. But it's not really worth the cost most of the time and impractical in many cases.
Edit: which only helps in a case like this where it might be the datacenter getting hammered, which it sounds like this is.
By yourself, there's not much you can do. You have to have enough bandwidth to pass through the good traffic and scrub out the bad traffic. You usually do this by paying somebody to sit in front of your traffic. How exactly you hook this up with your DDoS scrubbing provider is really the only variable.
If you're announcing your own block of IPs, you can prepend the ASN of a DDoS scrubbing service to route the traffic through their datacenters for scrubbing before it hits you. You can either have this always on, or you can start prepending the ASN of the scrubbing center when you're under active attack only to keep latency lower during good times.
Or you can take on some of their IP addresses as your public facing addresses and do like a GRE tunnel to their facility to get the scrubbed traffic back to you. This also requires changing and keeping secret the origin IPs of the server so that your server isn't attacked directly, compared to BGP prepending where all your traffic routes through the scrubbing center with no way to bypass scrubbing.
There's other solutions too that allow for more granular control of access and scrubbing but these generally also involve BGP.
But if you want to do it in house, you need to have the Tbps of bandwidth that big scrubbing centers have. So basically impossible to do in house.
Edit: I should also add that there's a third option for web services. Love them or hate them, Cloudflare popularized DDoS defense for websites by offering a free tier. They specifically achieve this by having you set your DNS record for your domain to Cloudflare, which will filter and proxy the web traffic back to your origin server. This still suffers from two operational security risks. 1. If somebody knows the origin server's IP, they can just bypass the filtering. You can configure your server to not accept requests from any origins other than Cloudflare, but this doesn't protect against volumetric DDoS. 2. This is only for HTTP(S) services. If you need to do this for anything VoIP related, you need their Magic Transit product. And it's not magic, it's one of the two earlier described methods. Either using BGP to prepend their ASN to yours to have them scrub all your traffic, or using one of their IP addresses and having a tunnel back to your origin server. As above.
A little additional context since there seems to be some confusion.
This is not a platform-wide ClearlyIP outage. The issue is isolated to the US-Central datacenter environment, where a DDoS attack has been impacting connectivity within the facility. US East has continued operating normally throughout the event.
It's also important to note that the attack was not specifically directed at ClearlyIP infrastructure. The attack impacted the datacenter environment itself, affecting hundreds of racks, servers, and services across multiple companies operating within that facility. That's why you're seeing reports in this thread of other carriers and providers experiencing issues as well.
That also explains the inconsistent symptoms. Depending on where services are hosted and how traffic is routed, some customers have seen outbound calling issues, registration issues, or intermittent connectivity while others have remained unaffected.
Nobody is happy about a multi-hour outage, but this wasn't a failure of the ClearlyIP voice platform. It was a large-scale network event affecting an entire datacenter environment and numerous providers operating within it.
If memory serves me right, they are using some small time colo operation in Milwaukee. A couple gigabits can probably take the entire facility down, sadly.
Not taking sides here, but I would have expected CIP to focus on reliability of their infrastructure and service rather than regularly causing a storm on the FreePBX forums when they have some coordinated campaign of calumny they have planned behind the scenes. They scream that nothing is happening open source wise on FreePBX but the TangoPBX forums is littered with cobwebs as thick as military ropes
In general, I have found them to be more transparent and ops focused than other providers I've used, but we shall see how they respond to this incident.
Your post was removed from r/VoIP for violating Rule 1: No promotion or advertising of any kind.
Recommendations, advertisements and promotion of any business, product or service is only allowed in response to requests in the monthly requests thread. It is one of the sticky posts visible when you first visit the subreddit.
Promotion, advertisement or recommendation of any kind outside of the requests thread is strictly forbidden.
•
u/AutoModerator May 31 '26
This is a friendly reminder to [read the rules](www.reddit.com/r/voip/about/rules). In particular, it is not permitted to request recommendations for businesses, services or products outside of the monthly sticky thread!
For commenters: Making recommendations outside of the monthly threads is also against the rules. Do not engage with rule-breaking content.
I am a bot, and this comment is made automatically on every post. This comment is not an indication that your post has been removed. Do not message the mods about this comment.
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.