Uptime.com Blog
After Transformative Rebranding, Wells Fargo ATMs and Mobile Banking is Down
Wells Fargo customers trying to check their balances and grab their morning joe ran into a snafu, as mobile banking and ATMs appeared to be non-responsive. Employees at a Wells Fargo data center, however, woke up to an entirely different kind of nightmare.
This began with the bank acknowledging an error had occurred around 9 AM PT, with a single tweet:
We apologize to our customers who may be experiencing an issue with our online banking and mobile app. Thanks for your patience while we research this issue. If you are impacted, please check back here for updates.
— Wells Fargo (@WellsFargo) February 7, 2019
Within two hours, a Fox affiliate in Minnesota was carrying an unidentified employee’s report stating that the cause was a server fire. The bank maintains a data center in Shoreview, and a few employees called the station’s tip line with the news.
Local journalism works!
One reported an actual fire and said flames were put out, claiming there was no backup and power was lost. As if the words “no backup” did not inspire enough dread, another report suggested some of the backup power sources (generators) had been burned.
A thread on the SysAdmin Reddit (which may or may not have some truth to it, but does correspond with the time) has some further details. Supposed data center employees there claim that the building’s fire-suppression system was activated. They also paint a pretty bleak picture, with one saying that teams are dead in the water over a DNS issue.
One user spoke of an upcoming trip and expressed concern about withdrawing foreign currency. The response: “I wouldn’t expect money for days.” #Thoughtsandprayers
Around noon ET, the local fire department acknowledged the fire suppression system was activated but claimed there was no fire:
There was NOT a fire at the Wells Fargo facility in Shoreview, MN. They did have an activation of a fire suppression system that was triggered due to dust from construction.
— Lake Johanna Fire (@LakeJoFD) February 7, 2019
For its part, Wells Fargo also put out an official tweet about it that seemed to confirm the data center outage:
We’re experiencing system issues due to a power shutdown at one of our facilities, initiated after smoke was detected following routine maintenance. We’re working to restore services as soon as possible. We apologize for the inconvenience.
— Wells Fargo (@WellsFargo) February 7, 2019
Did the suppression systems wipe out the data center, or was this a DNS issue? Some chatter on Reddit suggests that Wells Fargo might have had an opportunity to update its internal security policies as well.
We doubt anything malicious occurred, but this is big news for a major bank. Mobile and ATM functionality has been gone for what amounts to an entire workday, and the outage is still ongoing. If the reports are true, functionality could be down for days and it’s unclear if in-person banking is a viable option at this point.
Uptime.com can monitor DNS for your domain or application, so you’re aware of any changes made to registration and kept in the loop with any updates. It’s an important safeguard to avoid what is a common and difficult to diagnose error. DNS errors don’t typically start fires, but the pressure is on nevertheless.
Were you affected by the outage?
Minute-by-minute Uptime checks.
Start your 14-day free trial with no credit card required at Uptime.com.
