SOC Analyst Diary: What Detection Engineering Really Looks Like
Earlier in this series
There is a version of SOC work that exists in vendor slides. Analysts in a dark room full of screens, hunting adversaries in real time.
Then there is the version I lived as a SOC analyst at a major telecom, monitoring and triaging incidents across a very large infrastructure. That version is mostly a queue. What follows is generalized on purpose — no client specifics, no confidential detail — but the shape of it will be familiar to anyone who has done the job.
The queue is the job
A shift starts with a handover and a queue. Alerts, ordered by severity that someone — a vendor, a rule author, a past version of your own team — assigned long before this particular alert fired.
The work is triage. For each alert: is this real, is it urgent, and what happens next. The honest ratio is uncomfortable. The overwhelming majority of alerts in any SOC I have seen or worked in are false positives or true-but-harmless. Not because the team is bad, but because that is what detection at scale produces. A rule tight enough to never fire falsely is usually tight enough to miss the real thing.
So the fundamental skill is not exotic threat knowledge. It is fast, accurate dismissal. Recognizing the alert you have dismissed forty times, confirming in seconds that this is instance forty-one, and moving on — while staying alert enough to notice when instance forty-one is actually different. That last part is the hard part, and it is where fatigue does its damage.
Alert fatigue is a design failure, not a character flaw
Alert fatigue gets discussed as if it were a stamina problem. It is not. It is the predictable output of a system that cries wolf.
When a rule fires fifty times a week and has never once been a real incident, the rational response — the response any human will converge on — is to stop looking closely. The dangerous day is the day the rule is finally right. Blaming the analyst for that miss is blaming the person for the system.
The fix is unglamorous: every noisy rule is a defect, and someone has to own fixing it. Which brings me to the part of the job I came to care about most.
Tuning is detection engineering
"Detection engineering" sounds like writing brilliant new rules. In practice, most of it is maintenance of existing ones.
The loop looks like this. An alert is noisy. You pull its history: how often it fires, how often it was real, what the false positives have in common. Usually they do have something in common — a subnet, a service account, a scheduled job. You adjust the logic to exclude the known-benign pattern without excluding the attack the rule exists to catch. You document why. You watch it for a couple of weeks. Then you do the next one.
That last clause — without excluding the attack — is where the engineering lives. It is easy to make a rule quiet. It is hard to make it quiet and still useful. Every exclusion is a bet that attackers will not arrive through the hole you just cut. Writing the exclusion narrowly, and writing down the reasoning, is the difference between tuning and slowly blinding yourself.
The other half of the discipline is retiring rules. A detection nobody has acted on in a year, built for a system you no longer run, is not free. It costs attention. Deleting it is real engineering work, and it is the work teams skip most.
The vendor gap
Every detection product I have worked with promised meaningful coverage out of the box. None delivered it out of the box, and I no longer think that is a scandal. It is just how it has to be.
Default rules are written for an average environment, and no environment is average. The vendor does not know that your backup job looks exactly like data exfiltration every night at two, or that an entire class of "impossible travel" alerts is your VPN concentrator doing its job. The first months with any new detection stack are spent teaching it what normal means locally. Buying the tool buys you a starting point, not a capability. The capability is the tuning, and the tuning is staff time that never appears on the invoice.
This is also where I saw automation pay off honestly. Part of my role involved working with engineering to automate pieces of the SOC toolchain, and the highest-value automation was never "AI finds the attacker." It was enrichment: attaching the context — host role, owner, recent alerts, ticket history — to the alert before a human sees it. Ten saved minutes per alert, hundreds of alerts, every day. That arithmetic beats any magic.
3am, honestly
The mythology of the night shift is the sophisticated attack. The reality of the night shift is ambiguity with nobody to share it with.
At 3am the alert is never clean. It is a medium-severity detection on a system you half know, with a pattern that is probably the maintenance window and possibly not. The people who could confirm it in thirty seconds are asleep. Your choice is to wake someone — expensive if you are wrong — or sit on it — expensive if you are right.
What gets you through that moment is not brilliance. It is the written procedure. This is why I am stubborn about playbooks being decision trees for the first fifteen minutes, written for the newest person on the team — the case I made in Incident Response Playbooks That Actually Get Used. The playbook exists precisely for the moment when judgment is at its most tired and the stakes make it matter.
What I tell people entering the job
Three things, and none of them are technical.
First: dismissing alerts well is a skill, not a failure to do "real" work. Most days, it is the real work.
Second: every annoyance is signal. The alert that wastes your time daily is a tuning ticket. The lookup you perform by hand for the tenth time is an automation ticket. Analysts who file those tickets become detection engineers. Analysts who just endure become burned out.
Third: the gap between the marketing and the queue is not evidence that security is fake. It is evidence that detection is a craft that lives in specific environments, done by people, iteratively. The tooling matters — and I will have more to say about what deploying that tooling looks like in radically different organizations — but the tooling has never once been the hard part.
The hard part is the queue, and the discipline of making tomorrow's queue slightly better than today's.