Certified Great Place To Work

I started this article at the top of the eighth hour of Severity 1 bridge calls where I have said a cumulative six words and spent most of that time waiting on the person who started the bridge to show up to their own Sev1 while watching or participating in the third bitchy argument about a one-slide presentation that is only read as a self-licking ice cream cone and excuse to grill SOC and CTI analysts by leadership (correction: fourth argument). This is not the first day of this nonsense and certainly not the last. Tomorrow likely will be the same. As was yesterday and so were the last two years, at least I think it's been two years.

Some background information to start: My information security career started in radio operations and COMSEC in the Marine Corps where young, pig-headed and naive 19 year old me got handed several far too much equipment, a several hundred page manual and told "Good luck. The last guy fucked it up." Three years later, zero incidents and less than zero support from my leadership I got out and decided to do something relaxing. In that theme, I was hired as a DFIR analyst at a state-wide healthcare provider (somehow) and thrown in to the deep end to learn how to do my job. As this happened a week after the Solarwinds kerfuffle, there was absolutely nothing occurring that complicated my job further and has been widely considered to be a calm point for healthcare and information security. I spent the next two years there, understaffed and overworked dealing with a constant flood of near-misses, collecting new hats at a pace usually associated with the TF2 playerbase, while not loving every minute, at least quite satisfied. Usually. The work was challenging with long days, frustrating calls, inter-departmental turf-wars aplenty. But hey, we were doing good work. Nobody died on our watch, we kept the systems our care providers depended on running and kept the various alleged bears, cats and pandas out of the network. The IT, and especially IS departments never made the news. Until three quarters of us laid off so that our CISO could get a cushy job at Accenture.

Following several months of job hunting and leaving another job due to ethical concerns, I took the a job my old team lead made an introduction for. SOC analyst job, a non-trivial pay hit, and night shift. It wasn't great, but it was a job. What the hell, how bad can it be? If you look at the raw stats, I'm better off here than I ever was at the hospital. My SOC goes through probably 10x the case load, has a larger headcount and a much larger digital and geographic footprint. Plus, if I screw up, nobody dies or is injured. It should be a nice upgrade, all things considered.

Now, two years later, my entire job consists of closing the same dozen or so alerts, writing that one-slide report once a week and sitting in calls saying nothing. Note: it isn't that I don't have anything to say, but rather that I've been told "Stop bothering, it won't change" and "Oh that's too complicated/different/[fill in the blank], I don't think we could do that." to any suggestion for long enough to take a hint. This organization does not value security, it does not value people, despite priding itself on being a "Great place to work-award winner (again)" and making everyone feel welcome. The company doggedly determined to beat down anyone who dares question the status quo, I suppose the honeycomb logo may be a bit too on the nose. We've insisted on taking an approach to risk management that can be best described as "If I can't see it, it can't hurt me", which puts the security team in the interesting position where we need to continue trying to secure the environment and make measurable improvements, but we cannot actually shift the footing of the organization. To do so would implicate our leadership as being something other than perfect. We can't have that. Leadership knows all and the risk register is complete.

We remain trapped in a cycle of attempting to make the metrics we monitor look better as that's how we prove we should still exist. So what do we measure? Case closure counts, mean time to acknowledge, mean time to close, case load per analyst and number of alerts generated. Therefore, the changes we make attempt to raise these numbers through shambling, halting measures, it doesn't really matter how as long as the number goes up. So more alert types are added, more data sources are added, an arbitrary target to reduce MTTR/MTTD by the end of the year is passed down from on high. We continue to say we'll attempt to fulfill those requirements. We could tune out known false positives, but then the detection engineering team generates fewer alerts and the analysts close fewer. Well that doesn't show monthly growth, does it? So no, we're not going to tune that EDR alert that fires in the hundreds when a CI pipeline runs every Thursday afternoon. That would make the numbers go down! We can't have that, now can we? So we'll show that we're doing something about the problem and tune out an alert for a month. Next month it spikes again, and we get to show that we're making progress on tuning alerts and adding visibility. The joke about a mathematician watching two economists pay $100 back and forth to slap each other, claiming they're increasing the GDP comes to mind. So where does this leave me? Where does this leave the rest of the SOC? It leaves us scrambling to close alerts under SLAs, write reports about these alerts in too little time, get screamed at or bullied when they're off by a letter or two and then say to each other "It's really not that bad. It's a good job. It'll get better soon." before getting pulled into another call to be harassed over the same mistake from another manager. Rinse, repeat, do it again the next week.

So what does someone do about an organization like this? When I started this article, I wanted to have some satisfying conclusion. I wanted to have some good answer to what someone should do in order to dig their team out of this mess. I asked myself this quite often for myself as well. However, I have regrettably come to the conclusion that there is not always an answer that any one of us can use. Could we drag this team back on track and get them focused on the important things? Sure, it would take some time, involve a lot of retraining and leadership buy-in. There could be a concerted push to work on minimizing the workload the team is under, push automation heavily, embed security and ops people in each other's teams to help open up communications and get people reading from the same playbook. We could build a culture where screaming at your juniors was treated as abhorrently as it should be. We could build a culture where mistakes, oversights and missteps are treated as opportunities for improvement and to remediate issues as they're found. We could build a team where these are possible. However, first you have to excise the rot from the team, and when that comes from the top, it reduces you to few options. Continue to stick it out and hope things get better, try and become the leadership and change the culture from the inside and the most realistic option:

Get the fuck out.

That's it. Sometimes the answer is to get the fuck out. This job, this field, whatever else is not worth your health, your relationships or your self-respect. In information security we like to think we do important, critical work. The people cutting our paychecks view us as a roadblock and a cost center. The truth is a sliding scale in between. What we are not, is individually indispensable, there will always be someone ready to take your spot. And your employer knows it. What we are, however, are human. Deserving of respect, kindness, and belonging. We may not (and I would argue should not) obtain these things from our jobs, but we can find these things in the companionship of those around us. The people that do respect you. The people that do treat you kindly. The people that do make you feel that you belong. Help where you can, take help when you need it (which is more than we let ourselves think) and extend that kindness to others where you can. However, when those around us attempt to strip those unalienable rights, there are two options: Stand and fight or leave. Unfortunately, the worker has next to no protections in the modern workplace and any fight will likely result in termination if done alone. A concerted push back can be done, but requires buy-in from all involved. However, outside of a few isolated people on my team, most went along to get along with their own harassment and perpetuate it onto others. At that point, you get the fuck out. While I don't recommend doing this blindly as I did, no job is worth your self-respect. No job is worth being degraded. No job is worth being miserable. If your employer thinks otherwise, they can go to hell. Find somewhere better, start your own thing with like-minded people like I am. Hell, go do something that makes you happy. But never, ever, compromise yourself for your employer.

Your employer will not spare you a second thought when cuts happen.

Why give them the same courtesy?

Fuck em. Leave. What's the worst they can do? Fire you?