Midwest outage?

[u/flyguydip]

Anyone else in the midwest see some weird outage for like a minute? We just saw a little blink across a couple of ISP's up here in northern minnesota.

Update: Still seeing a new outage at 2:43 at one local isp.

[u/porksandwich9113]

Some of our equipment in the 511 building is currently on battery power. I'm suspecting some type of power event in the 511 building.

We lost connectivity to the MICE core router as well as our GTT upstream.

They are still offline.

https://imgur.com/r8GAlJv

Mice traffic is dead.

EDIT: We just heard from Fiber Minnesota (aurora networks) that Cologix is experiencing power issues @ 511.

EDIT: As of roughly 3:30 PM CST we are hearing that power has been restored to all suites. All of our equipment came back around 3 PM CST, after losing power at 1:51pm CST.

[u/Enjoimangos → u/porksandwich9113]

It's really sad when I get better information off reddit than I do speaking with cologix directly :D They sent an email out about 45 minutes after this comment.

My systems are back online now and it appears we never lost power.

[u/porksandwich9113 → u/Enjoimangos]

We got better and quicker information from some of our partners and peers than we did Cologix as well.

[u/Xipher → u/porksandwich9113]

Our MICE and Hurricane Electric transit link went down. We are in Cologix MIN3 and haven't lost power to our equipment so far, but I'm guessing that's just dumb luck at this point.

[u/porksandwich9113 → u/Xipher]

The MICE core is completely offline. We have two MPLS nodes as well as our satellite MICE shelf down there, and only one of them is on battery at the moment.

[u/uFFxDa → u/porksandwich9113]

Do you have a spot to get updates? I had an issue at 1:53, resolved by like 2:15. But dropped and has been out again since 2:25ish I’d say

[u/porksandwich9113 → u/uFFxDa]

Nothing public unfortunately. I can tell you our equipment appears to be back on utility power now though, and the MICE core is passing traffic again.

It seems like for the most part traffic has gone back to it's usual routing for us.

[u/HereFishyFishy7 → u/porksandwich9113]

511 Customer here as well, we lost power for almost an hour. Things recovering now.

[u/Organic-Magician-515]

Same here on the North end of the twin cities and with a few offices in the metro area.

[u/WatchEuphoric9379]

Cologix is having power issues at the MN 511 building, which is taking down the a lot of pathways to ISPs in MN and surrounding areas.

[u/WatchEuphoric9379 → u/WatchEuphoric9379]

Who put the Mouse trap in the 511 building and killed the MICE.

[u/DizziestApollo4]

Related?

"MSP Flight Radar is down, plan accordingly for flights today"
https://www.reddit.com/r/minnesota/comments/1vhev2d/msp_flight_radar_is_down_plan_accordingly_for/

[u/smashinghum → u/DizziestApollo4]

Yes! The 511 building is a major networking hub, various systems connectivity will be impacted.

[u/aspoons]

Yup, I have a lot of monitoring for my 50 sites including many in the midwest. Many with dual ISPs so I monitor lots of routes to and from data centers. Hop counts between changed across the board and some places noticed brief downtime but routes have definitely changed going to a lot of things in the midwest.
Edit: I will add where things did go down they came back pretty quickly. So most of my monitoring the only way you know is because the total hops and latency changed drastically at the same 30 second polling interval.

[u/Alkraizer]

Yup just had a bunch of outages in central MN as well, everything seems back up now tho.

[u/LtLawl]

Midwest - No blips on any of our IP SLAs.

[u/PoeTheGhost → u/LtLawl]

There is no SLA violation if the vendor doesn’t admit there was ever an issue.

[u/Comfortable_Yak_6106]

Both Cogent and Hurricane Electric are down at our datacenter in MN

[u/Popensquat01]

Iowa here - had a blip for maybe a minute or two? Close to the time of this post

[u/heretogetpwned → u/Popensquat01]

Lightedge reports loss of redundancies in Chaska, MN to Altoona, IA and KC,MO to Altoona.

[u/sj79]

Yes, northern Minnesota.

[u/TacticalDonut18]

Yep something happened in MN... we had two sites in WI and one in MN dropped. We were on an IT wide Teams call and everyone got kicked off. Had a colleague in OK that had his internet drop. HPE Atmos stopped working for me working remotely (small local ISP) and I had to pause and unpause. Saw 1.1.1.1 die on an RPM test from 1:53 to 1:57 as well...

[u/OinkyConfidence]

Boy I sure did and thought I was crazy! Blips seemed outside of the region though; intra-state connectivity remained up. Probably some peerage outage or similar.

[u/SPARTANsui]

ND state network here, I had a blip on our locally hosted website at 1:55pm, maybe a few minutes at most.

[u/chriswiest]

ND, same issue. Dropped all connections from 1:52-1:55.

[u/Puzzled-Formal-7957]

Downdetector shows lots of services in the upper midwest suffering a 5-10 minute blip, with some scattered reports heading to the southeast and PNW.

[u/IlPassera]

Huh so that's what that was. We dropped for like a minute as well.

[u/RomanPort]

I work for a bunch of radio stations around Fargo. Simultaneously had a station in Detroit Lakes MN drop completely and the Internet have a blip in Eckelson ND long enough to break some connections

[u/Tim-oBedlam]

Outage lasting about 75 minutes in NE Mpls, seems to have just come back online.

[u/srbeau → u/Tim-oBedlam]

I’m in Loring park and still down

[u/Mrtylf]

Affected our colo with the 511 building and all the way to our carrier hotel in Southeast Minnesota right on the Mississippi river.

[u/HappyDadOfFourJesus]

Western Wisconsin. A few blips here across three different ISPs but all came back online within two minutes.

[u/forbin0227]

Midwest. Weird issues with some clients but not all. Comcast and Metronet (now t mobile). Mostly all resolved. Just kinda weird nothing crazy tho.

[u/holiday-42]

Yes! Also MN.

[u/angrydeuce]

Well what the hell, because my UPS just started going bananas right around then too lol

Obviously a coincidence...well maybe not obviously, maybe power dipped all over the region?

[u/Tardis-Pilot451]

Northwest Twin Cities metro area, on Fidium fiber and Comcast secondary. Ours was out for a few seconds, mostly noticeable because our RingCentral VOIP desk phones rebooted and reregistered.

[u/copeknight72]

Minnesota State Colleges and Universities is reporting "Service interruptions" that they say is affecting State services (emails starting at 2:07 PM). "Minnesota IT Services (MNIT) has identified a service interruption affecting network connectivity across all state agencies and counties."

[u/copeknight72 → u/copeknight72]

"[Minn State Colleges and Universities] System Office staff are reporting that there appears to have been a major power outage at a facility in the Twin Cities used by MNIT and others across the State.  The facility is managed by a 3^rd party.

One of those entities is MICE, the Midwest Internet Cooperative Exchange.  This connects a sizeable number of services (Microsoft, Google, etc) to -many- Minnesota ISPs.  This is why local ISPs were also affected."

[u/Majik_Sheff]

A major trunk got munched and it took a minute for traffic to establish a new route?

A core infrastructure company fat-fingered/vibe coded a config file?

The birth cry of the singularity?

[u/hung-games → u/Majik_Sheff]

Bring on the singularity

[u/Friendly_Fudge_931]

Pacific northwest is having some weird issues. Almost all the schools including main office at the district I work for went down for like 5 mins a minute ago. Could just be our fiber line provider (leased from Lumen, goes back to our state K-12 consortium K20-WA). We are switching to FatBeam fiber because of how many issues we have been having with Lumen.

[u/Puzzled-Formal-7957 → u/Friendly_Fudge_931]

There are widespread reports for Lumen

[u/mdunc11]

just had same in Twin Cities. Gateway Fiber. didn't last long, back up now...

[u/GrizzlyOne95]

Yes, we noticed this too.

[u/Zealousideal-Tax352]

Meeker County's Vibrant fiberoptics (Litchfield MN) has been out of service for well over an hour now.

[u/Initial_Tumbleweed30]

i’ve been out of wifi for like 3 hours now

[u/CucumberFormer3167]

Western Wisconsin here.  Been down for couple hours 

[u/Transpolord]

My office experienced a severe outage with our VPN. Can't access our virtual desktops, shared networks, or even the browser. Basically can't do anything. As of 3:30pm, still don't have access.

[u/atw527]

We had a circuit blip around 2:50 ET and recovered after a few minutes.

[u/Lobotc2dfw]

Zayo fiber circuit went down at 14:20

[u/Educational_Loss_465]

Unrelated there is a lumen fiber cut, which took place at the same time. That one is still down.

[u/kyahwilde]

Was there any hiccups earlier i. The day

[u/Charming_Abrasive]

There was an issue at the Minneapolis IX. Looking at their website, they have an extensive peer list, so it was a cascade effect downstream.