Hacked! Solve the Dreaded DevOps Problem With This
Hacks that make headlines are painful for everyone involved, but with some clever preparation and web monitoring at your side you can avoid the worst of this pain.
Those who have been victimized face a steep uphill battle to reclaim trust and authority. Unwitting victims, like customers and end users, suffer downtime or leaks containing personally identifiable information. If your eye is not on security, your organization is inviting these kinds of attacks.
At Uptime.com, we deal with organizations that have IT staff sizes across the gamut. Everyone of them feels the same sting when a hacker targets them. Today, we’re going to ask ourselves what we can learn from these big name attacks.
System Complexity Makes it Easy to Hide Attacks
Researching this post, we came across an interesting question: how has the complexity of networks changed threat response? Are complex networks more or less difficult to manage than their older counterparts?
The answer is yes, IT infrastructure has undoubtedly grown in complexity but staff is no more or less equipped to monitor its intricacies than at any other point in computing history. Sure, maybe the tools we relied on are different, but the symptoms are the same.
Let’s pick this apart a bit.
Learn Your Infrastructure
Are complex networks more or less difficult to manage? Let’s answer this question with a question:
Do you know how much infrastructure you manage? That’s a pretty solid litmus test to determine whether you might have a vulnerability. And you might be thinking you know everything under your purview, but what about old infrastructure from before your time? What about new stuff coming down the pipeline?
Staff Size and Infrastructure
What your network consists of has likely changed. You know what probably hasn’t? The size of your staff. Consider changing that.
This #flashbackfriday we’re moving into the 90s!
By 1992 all library employees had a desktop computer and were linked via a local area network. Two of these photos show our staff being trained to use the computers and the network, while the other shows reference area computers. pic.twitter.com/s5xKx7nqYY
— Laupus Library (@eculaupus) September 20, 2019
Yes, I do get how smug that sounds, but these attacks cripple infrastructure. They leak personal data. They lead to lawsuits. Which costs you more? A new hire, or a lawsuit?
Smaller departments usually do not have the staff and resources needed to both bug hunt and build. Smaller IT departments often need to balance many tasks, like maintenance or development, while worrying about uptime.
Enterprise can have the opposite problem, where a rush to get new releases out can lead to contracted development without proper oversight. Ideally your staff catches bugs and thoroughly tests everything under release. But let’s be honest: there are some things that just slip through the cracks as an organization grows in scale.
A cost effective approach to both of these problems is automated monitoring.
It’s very tempting to consider uptime an afterthought when things are built well. Especially when teams find themselves vibing. Maybe code feels tight and meetings are productive. It’s in these “it just works” scenarios where disaster can sneak up on you.
Automation Catches Things Casual Observers Won’t
Most IT departments are not prepared for a true intrusion because there are simply not enough eyes on the problem. Departments may not be as well-funded as they need to be, and there is a chance routine sysadmin work just overloads the few people who are in charge.
A proper infosec analyst looking at this would first wonder why there is HTTP traffic related to .NET assemblies at all (this sounds weird). Then comparing two requests or so would likely show a suspicious pattern of all the hex changing randomly while the rest of the payload is cookie cutter the same.
This commenter went on to say most IT departments just can’t do it. That is not a majority of organizations and it would be very easy to miss something like this.
We believe the best approach is thoughtful automation. Thoughtful automation looks at how monitoring can cover when you cannot be around. It handles the heavy lifting of detection and diagnosis so you can focus on growth and development.
First response capabilities give you a head start on diagnosing the issue.
An Automation Use Case
As an Uptime.com user, you have access to our REST API. Using this system, you can bake check creation into your development pipeline. The RUM endpoint will help you create a check to track real user sessions, which lead to performance insights and usability improvements. The HTTP(S) endpoint is the backbone of your monitoring, with one-minute intervals and a sensitivity of two to ensure downtime is legitimate.
Sensitivity settings are important because they ensure the outage is not location based, or a false flag. In general, multiple locations experiencing an outage point to legitimate downtime. This is why external web monitoring is most effective when you are taking a proactive approach to downtime.
Information Can Be Scattered
Another issue with attacks of magnitude is the information available to the public. During, and just after an attack, lots of analysis can mean bad information getting mixed in with the good. Even your best most trusted analyzers can get it wrong. Take the time to examine the available evidence from your own network. You can take what a blogger says with a grain of salt, but your network data speaks the truth.
Alerts only do good to your organization if there is some semblance of logic behind them. What are you monitoring, and is it mission critical? If it is mission critical, have you adequately documented processes to get it back online if it fails? If you have not, do you have trained technical personnel ready to step in and take over?
Protip: Uptime.com users can include notes with each check, which may include runbook instructions for your personnel to reboot a service or diagnose further.
Escalations can help eliminate prolonged downtime. Escalations are great. They give juniors experience managing the incident with a pre-ordained amount of leeway ensuring nothing spirals out of control. Post mortems are good practice after an escalation occurred: what did the senior do that the junior either did not or could not?
You Can Be Caught in an Attack Not Meant for You
The last take away from hacks of scale is the idea that you may not be the intended target. Not everything revolves around you, in this case for the better. But it still places the burden of a proactive defense on your organization.
Your department needs to critically think about its approach to the doomsday scenario, and should start mapping out network infrastructure and setting automated monitoring in place. Downtime skyrocketed in 2020, and 2021 could be an even greater challenge without proper automated monitoring.
Minute-by-minute Uptime checks.
Start your 21-day free trial with no credit card required at Uptime.com.