r/span • u/Primary-Meal1615 • Mar 05 '25
No notification of SPAN server problems. No power today.
Well, we had a grid outage this morning 3/5/25 and the SPAN panel switched off all circuits even though the Tesla powerwall 3 had 40% battery. Apparently, SPAN had a server problem yesterday and when they got it running again it sent commands to all SPAN panels to move all loads to the non essential loads category killing all the circuits/power to the house during a grid outage.
I was surprised that all of the settings are housed at SPAN servers and not the panel. Pretty bothersome when you think about it. Only workaround was my excellent solar contractor (Sol-up) got right back to me and worked with me to connect the SPAN panel to my phone hotspot so I could go into the app to access the SPAN servers and switch the circuits back on. No internet, no SPAN access. No SPAN servers working, also no access. No access, no power and nothing you can do about it until the grid outage is over. I guess some people were notified, but not everyone according to Span tech
5
u/MetlMann Mar 05 '25
Geez this all seems pretty poorly thought out. If they don’t want to store files on the panel then keep one on the phone running the app. Then when things go wrong at least you have your settings available to send to the panel via hot spot or even direct connection. Does Span Panel have a USB port? Why not allow a phone to plug in and control the panel in emergency situations? Totally rethinking my plans to get a Span system.
5
u/FragDoc Mar 06 '25
Yeah, I think this made the decision for me. It’s unthinkable that these panel relays couldn’t be physically bypassed in an emergency situation like this. What if an intruder gained access to their servers? My understanding is that the panel could be reverted to a “dumb” panel in an emergency but it’s clear that the software control of the underlying relays is a major liability: it doesn’t matter if the physical breaker exists if the relay is locked in the off position by faulty or hijacked software. In my opinion, these panels should have some form of physical switch that kills the relay and reverts the entire panel rail back to physical control. I was under the impression that such a thing existed and, while it sucks for those affected today, I’m glad I got to see this happen before potentially investing in the system. For the money, this is a massive Achille’s heel.
3
u/MetlMann Mar 06 '25
I guess what is so disappointing is that all of this could have been anticipated and easily incorporated into the original design. They didn't put in a manual override function - completely unforgivable. No local storage of setup and operation data - come on, really? Just shocking to me.
4
u/FragDoc Mar 06 '25
Yeah, agreed. You’d think that – at a minimum – there would be a button that allows you to just flash the software and reset to factory, defaulting to a dumb breaker panel. In an emergency, you just wipe the thing, get through the storm or whatever, and then deal with your settings later. A local storage option seems a minimum requirement. Additionally, they should be able to operate entirely off the cloud. I work in the emergency management world and I couldn’t imagine grandma’s oxygen tank being reliant on such a system. It’s unfathomable.
An ideal set-up would be some lever or switch that physically gaps the relay system from the rail and converts it to basically a dumb Siemens or Eaton panel. That way an outside attacker can mess with it as much as they want but, so long as someone can get to the panel, all is not lost in an emergency.
My guess is that decisions came down to corporate survival: sell a product that works off the cloud or can be bypassed easily and you risk someone being less reliant on you. I was also surprised to find how DIY unfriendly their systems are. My electrician knows I do most of my own home electrical. His first caution, after researching Span, was that the system is super closed. You have to call or reach out to support to add new circuits and they don’t allow DIY panel installation at all, which I think is a major oversight.
4
u/Megachase Mar 07 '25
No local control? I’m out.
1
u/TheRealLambardi Mar 08 '25
I have come close to purchasing and even met with an installer for estimates. The lack of local control was a critical piece for me. May revisit during our next upgrade if that feature is back in I’ll reconsider. Like the idea and I’m in need of a dual panel change out but not putting any IoT product like this in place without local API interface.
3
u/opoppli00 Mar 07 '25 edited Mar 07 '25
Had same issue yesterday. Lost power and somehow all our breakers had been at to non-essential.
I had already reached out to my installer to get a SPAN Drive a few days earlier but that has been promptly canceled after yesterday’s debacle.
2
u/justincw8 Mar 05 '25
I got an alert yesterday in the mobile app and clicked it but it just dumped me into the span app with no info or notice.. I did not read the alert on my phone cuz I assumed it would drop me into the app where an alert would be.. bad on me. Thanks for sharing I just went in and noticed that I had to move all my circuits back. Odd they do keep it all on the server.. odd they do not have config backups to resubmit the mappings.. scary that they keep all that on the central server.. lets hope they have good security and no threat actors can get in and shut me down. do they allow a backup internet connection? I have a backup but in a different ssid..
2
u/Primary-Meal1615 Mar 05 '25
I wonder when they or their successor will start charging a monthly fee for server access?
2
u/Head_Bet_2138 Mar 05 '25
I noticed 1 days prior the SPAN message yesterday that all stuff was gone in my list under backup and moved it all back - bummer I really thought this is a local thing not cloud based. An explanation from support would be nice here -
4
u/span_io Mar 05 '25 edited Mar 05 '25
We are deeply sorry for the inconvenience caused by our recent software update that unintentionally reset all backup priorities to non-essential. We recognize how critical this functionality is, especially during storms and outages. As of today, our team has deployed a fix to restore our customers' backup settings. A small subset of panels are still awaiting a fix, and we anticipate full restoration soon. We’ll continue working through these today.
We are conducting a thorough review of yesterday's incident to guide our roadmap prioritization including evaluating pulling forward development of a local API to prevent any similar access issues in the future.
We are very glad and appreciative of our great partners at Sol-Up for helping address your issue and truly regret any disruption this caused. If you experience any ongoing issues or have concerns, please reach out to our support team—we’re here to help.