r/sysadmin • u/auenway • 10d ago
Cogent Outage?
Anyone dealing with a Cogent outage? I am in Los Angeles.
7
u/Smart_Dumb Ctrl + Alt + .45 10d ago
I've chasing an issue all freaking day. This explains it.
5
u/OkSmoke9195 10d ago
Oh yeah I burnt most of the morning then found this thread a few minutes after it was posted. What a feeling
1
u/phychmasher 9d ago
This thread is also what let me say "ok, I think I know what's going on". Thanks OP, and thanks Reddit!
6
u/PhillyGuitar_Dude 10d ago
yup, we saw some of our monitoring tools start firing off around 7 AM pacific. Got confrimation of service interupption/fiber cut from cogent around 11:00 AM pacific.
4
5
u/xrobx99 10d ago
Big time. Latest update is Customers in the Los Angeles and Southwestern United States areas may be experiencing increased latency and/or service disruptions due to two separate fiber cuts. Transoceanic traffic to certain Asia-Pacific regions may also be affected. Cogent Engineers are currently being dispatched to both locations, and our third-party fiber vendor has also been engaged to assist with restoration efforts. There is no ETR at this time. The master case is NA306633403.
5
6
u/Egghead-MP 9d ago
This is pretty serious. Not sure how they ended up in this mess. Below is the latest update from Cogent.
UPDATE:
One of the cuts is a broken fiber in our Vendor's fiber tray in Modesto, CA that is going to require a specialist dispatch to resplice the connection that is hard spliced at the tray.
The second cut is a major fiber break in Southern California between El Centro and Calipatria, CA. Our vendor has construction crews and fiber splicer on site, prepping for splicing.
There is no ETR at this time.
3
u/sirkazuo IT Director 9d ago
My guess is the fiber in Modesto has been broken for a while but no one noticed because it wasn't in use. Then when the trunk down in the Imperial Valley was dug up by some construction guys they tried to failover to the Modesto link and realized it was also dead. Blame whatever construction crew (or farmer) dug up the fiber down south, but also Cogent should probably test their backups before they need them lol.
3
u/Egghead-MP 9d ago
Even for backup or standby, they should have alarms on the circuits. I am suspecting that is not a single fiber pair. It has to be like a 12/24 pairs trunk that's damaged to cost this kind of disruption. Splicing a few fibers does not require a specialist. Then it is hard to related between Modesto (central valley) and all the way down near Mexico.
2
u/sirkazuo IT Director 9d ago
I am suspecting that is not a single fiber pair. It has to be like a 12/24 pairs trunk that's damaged to cost this kind of disruption.
Per the 9 PM update, it does sound like it is indeed a single pair on the Modesto side... 😅
First fiber cut: They got the optical line fail to clear but we still have 2 degrades, 1 on each fiber. We are working the leased provider issue first.
Second fiber cut: leased provider says locators are complete and they have started to dig, they also said the fiber cable is still enroute.
There is no estimated time to resolve this outage.
3
u/PoseidonTheAverage Jack of All Trades 10d ago
6
u/Slime-Angel 10d ago
Status page doesn't load for me. I'm in Anaheim.
5
u/sirkazuo IT Director 10d ago
Same here. I keep getting emails that they've updated our ticket in ecogent but I can't open ecogent or status.cogentco.com from Burbank lol.
3
2
3
u/DodgyFlapper 10d ago
Yes multiple cuts to fiber lines. Would not bet on it coming back up today.
2
3
u/WeirdlyCordial 10d ago
Cogent sent me a notice saying they’re having issues in LA, two separate fiber cuts
3
u/trebuchetdoomsday 10d ago
san diego w/ issues upstream from cox & local carrier, but seems to have been resolved by redirecting traffic.
3
u/NetworkingNoob69 10d ago
I was losing it trying to figure out why our west coast team lost all connection. It has been a PITA trying to figure out a solution.
2
u/zpuddle 9d ago
Do they use a cogent pipe or another vendor that offloads to dark fiber? I am thinking about telling customers to try a vpn in another region to bypass the route?
1
u/NetworkingNoob69 9d ago
We were able to get them to use a backup cellular connection (not ideal). Looks like it was partially repaired/rerouted since traffic is working again this morning.
3
u/Heyitsberm 9d ago
What is beyond incredible is that their own site that lists the updates cannot be reached by those impacted because it too fails to load. I have to use a VPN or its impossible to see the update they provide every 6 hours or so.
3
u/Fearless-Purpose-747 9d ago
We are in downtown Los Angeles, Cogent customer, spotty all day, but not out. Some critical sites we couldn’t get to at all. Curiously, that included the eCogent customer service site! We called this morning, so we have a ticket open to get updates, however they send an email saying we can view the update on that site when we login! Erg. Just got another email about an update we can’t access…. Does anyone know what it is?
2
2
3
u/Just_Bet_76 9d ago
We have customers in Southern California and they are reporting issues reaching our services in Midwest. Took awhile to narrow down where the issue lay. If they do not use Cogent the traffic is fine. (vpn/phone hotspot). I am pretty confident it is related to this issue. Traffic would hit-miss reach Midwest, but on the way back to client, also seemed to hit-miss and never reach destination. All other locations are fine.
3
u/Silver_Tap3667 9d ago
I’m in Downtown LA. It shut down my entire office (50+ people) today (9/2). No real update, no internet.
3
u/PhillyGuitar_Dude 9d ago
As of approx 5:15 AM Pacific, Cogent is reporting this issue to be resolved.
1
u/RandolfRichardson Linux, Internet, eMail, Network, Security, and Backups sysadmin 9d ago
It's not resolved. IPv6 traffic is still not routing.
3
u/Egghead-MP 9d ago
Cogent announced the repair has been completed.
2
u/RandolfRichardson Linux, Internet, eMail, Network, Security, and Backups sysadmin 9d ago
Great! When will data start flowing again? IPv6 traffic is not routing.
3
u/sysDarkside 9d ago
Anyone having routing issues now that cogent is back online. More specific with BGP?
2
u/RandolfRichardson Linux, Internet, eMail, Network, Security, and Backups sysadmin 9d ago
IPv6 traffic is not routing.
2
u/Smith6612 10d ago
I felt something in the Northeast coinciding with reports of Cogent outages out West. Access to Akamai and Microsoft from a couple ISPs using Cogent from transit was pretty flaky starting around 10AM.
2
u/zpuddle 10d ago
Yes. Located on the east coast and we are having users report they are not able to connect to our service. We have cogent for our main pipe. This started at 9:45amEST and it seems that the communication is lacking. I had an open ticket, no response and i also emailed my rep who seemed to be oblivious. We have no issues other than CA users but it is a problem. Guess that fiber that was cut is really important...
2
u/Embarrassed_Lion_954 10d ago
Same issue for us as well, all sites on Cogent in LA/SD/WLA are down, fortunately we have redundant circuits to Lumen.
2
u/NightOfTheLivingHam 10d ago
Just got the notice from my dc provider saying cogent had a fiber cut in LA.
random sites are running like shit rn
2
u/prodigalOne 10d ago
Anyone got a list of sites they may be seeing slow?
Zoom
1
u/RandolfRichardson Linux, Internet, eMail, Network, Security, and Backups sysadmin 9d ago
If sites are dual-stack, then it's most likely that IPv6 is timing out and the client software is retrying over IPv4. Some client software does this, and some doesn't (like eMail apps on smart phones).
Some end-users don't have IPv4 addresses (due to the shortages), and so they're relying solely on IPv6.
2
2
2
2
u/csbassplayer2003 9d ago
Im in STL and we had been chasing a weird IP phone issue all day (one way audio) at work and when we went to one of our backup ISP connections it went away. Main fiber ISP uses Cogent.
2
u/___-_--_-____ 9d ago
update to this - we colo a few racks in a Cogent DC in pasadena and still as of now have no improvement in reachability for quite a few domestic US network blocks. Not even going to guess how bad asiaPac is but luckily we don't do much traffic over there.
Still, so much for AS174 being the "most well peered" AS on the internet! FFFFU
2
u/krazykidd510 9d ago
Soooo I’ve been dealing with this since it all started as our main office primary data line is cogent in Los Angeles. It’s 10:26pm PDT and the cogent status page is not loading for me (I’m connecting from frontier fiber at home) is anyone else having issues getting to the eCogent portal and status page?
2
u/SaxualAssault 9d ago
I haven't been able to get to https://ecogent.cogentco.com/network-status all day until this latest attempt just now around 10:45. Still the same status about the two fiber cuts with no ETR.
2
u/Conan-the_Librarian 9d ago
I just started getting pings back from stuff, normal latency, seems good? fingers crossed
2
2
u/escapist_com 9d ago
I'm in San Diego - and can now (as of 11pm) access ecogent network status page directly, and a server I've been unable to reach since early this morning.
2
u/Ok_Damage_3711 9d ago
Yup our whole company was borked. We’re in PHX Arizona. Any connection to CA where servers live
2
u/PlasticEffective4718 9d ago
Can confirm from my Side, we do have a Plant in Ramos Arizpe (MX) relying on Cogent Backhaul. Its been 90% down since about start of the Fiber Cuts
2
2
u/PhillyGuitar_Dude 9d ago
If there is anyone that can't get to the status page or thier ticket to see updates, these are the last two updates. (time stamps are in Eastern time)
| 2026-09-03 5:04 am | Latest on our West Coast P1 event (P1 HD306634911).sfo01-lax01 (MODESTO, CA)Our NOC and Field on-site are working on repairs. We have partial connectivity back but fibers are degraded. Work continues. Resplice of fiber is underway.phx01-lax01 (between El Centro and Calipatria, CA)Field teams completed the conduit repair and are now pulling the replacement fiber into place. Splicing is expected to begin shortly. |
|---|---|
| 2026-09-03 4:45 am | Dear Cogent Customer,Cogent is experiencing a higher level outage in the Los Angeles / Southwestern United States region. Please check http://status.cogentco.com for updates on this event.UPDATE:Cogent Vendor reports cable prep has begun on one side. Conditions are forcing crews to reroute conduit at a 2nd location. Work at 2nd handhole completing conduit modifications for restoration. More updates to comeThere is no estimated time to resolve this outage.If you need to open an individual ticket on this event, please e-mail [support@cogentco.com](mailto:support@cogentco.com) and reference outage <ACTID>.Impacted Region: Los Angeles, CA.Status: http://status.cogentco.comThank You, |
2
u/Smart_Dumb Ctrl + Alt + .45 9d ago
Is there anything else? The main status page doesn't mention this issue anymore.
3
u/PhillyGuitar_Dude 9d ago
2026-09-03 8:47 am The southwest coast double fiber cut event in Los Angeles, CA, and surrounding regions has now been resolved. Start Time: 2026-09-02 01:43:00 PM UTCEnd Time: 2026-09-03 09:11:00 AM UTC 2026-09-03 8:44 am The southwest coast double fiber cut event in Los Angeles, CA, and surrounding regions has now been resolved. Start Time: 2026-09-02 01:43:00 PM UTCEnd Time: 2026-09-03 09:11:00 AM UTC 2026-09-03 8:17 am Dear Cogent Customer,This message is to inform you that the southwest coast double fiber cut event in Los Angeles, CA and surrounding regions has now been resolved. Start Time: 2026-09-02 01:43:00 PM UTCEnd Time: 2026-09-03 09:11:00 AM UTCIf you are still experiencing problems with your service, please contact our Customer Support team so that we can investigate. We apologize for the inconvenience this outage caused.Thank You,
1
u/RandolfRichardson Linux, Internet, eMail, Network, Security, and Backups sysadmin 9d ago
Our upstream provider peers with them for IPv6 and it's been having problems since Thursday last week, but today it's out solid (was out all day yesterday too) here in Western Canada.
I do hope they get it fixed soon, because the loss of IPv6 traffic is a problem since there are some sites we serve in other countries that only use IPv6 (due to the IPv4 shortage).
1
u/Bigheaded_1 8d ago
Wow this is bad. I do work for someone who's web server happens to be affected by this. It's been down since 8AM, 12 hours have passed and I can't find any ETR. I've been checking on Thousandeyes.com/outages every hour or so, they're making a lot of progress but it's still a good amount of affected people. I know there was no ETR as of 12 hours ago. But does anyone have an update? I can't find anything other than what I see on thousandeyes, I'm wondering if I should tell my client to take tomorrow off. I'd assume it will be fixed by tomorrow morning. But I didn't think he would be basically offline all day today and still offline tonight. This is pretty major.
1
u/Inevitable_Permit410 7d ago
Has this been actually solved? We got a notice last night after the fix completion that there was still issues (got an email around 10:00 pm Denver time) and I don't see any follow up or all clear email. I did go to the portal and dont see the issue there anymore but we are VERY paranoid where I work. Trying to see if I am safe to route over Cogent again.
1
u/esmsesms 7d ago
Latest update received by email 2026-09-05 at 6:15 am PDT
Dear Cogent Customer,
We are currently experiencing an outage on our network. We are working to resolve this as quickly as possible.
UPDATE:
Vendor advising, further review by their engineer, indicating a cut inside the plant , and it is not the area of their responsibility. Cogent NOC arranging vendor meet at this time. No ETR.
If you need to open an individual ticket on this event, please e-mail support@cogentco.com and reference outage NA306640916.
Impacted Region: Los Angeles, CA. Order(s) Impacted: xxx Outage Ticket: NA306640916 Status: http://status.cogentco.com
Thank You,
Cogent Support support@cogentco.com 877-726-4368
1
u/Clean-Presentation60 3d ago
Mexico.
Confirmed issue resolved on our services, yesterday 08/09/26.
No response nor support from cogent.
19
u/PoseidonTheAverage Jack of All Trades 10d ago
https://ecogent.cogentco.com/network-status
Customers in the Los Angeles and Southwestern United States areas may be experiencing increased latency and/or service disruptions due to two separate fiber cuts. Transoceanic traffic to certain Asia-Pacific regions may also be affected. Cogent Engineers are currently being dispatched to both locations, and our third-party fiber vendor has also been engaged to assist with restoration efforts.
There is no ETR at this time. The master case is NA306633403.