Detection: Signature, Heuristic and Behavioural

Heuristic detection: catching the similar

Heuristic detection identifies likely malware by suspicious characteristics or similarity to known malware, rather than by an exact signature. Instead of asking "is this exactly a known sample?", it asks "does this look like malware?", examining the code for traits common to malware or for resemblance to known families.

Its properties:

• Catches variants and the similar-to-known. Because it does not need an exact match, it can flag a modified version of known malware, or a new sample sharing malware-like traits, which signatures miss. This closes part of signatures' blindness.
• Risks false positives. Judging by resemblance rather than certainty, it can flag legitimate programs that happen to share suspicious traits, so heuristic detection produces false alarms that signature detection largely avoids. Tuning the sensitivity trades detection against false positives.
• Still fundamentally about inspecting the code (what it looks like), so a sufficiently different or well-disguised new malware can still evade it.

So heuristic detection extends reach to the similar-to-known at the cost of certainty, and it is the middle method: broader than signatures, less precise, and still ultimately looking at the code rather than the behaviour.

Behavioural detection: catching the unknown by what it does

Behavioural detection takes the decisive different approach: instead of examining what a program looks like, it watches what a program does while it runs, and flags malicious behaviour, regardless of whether the program has ever been seen before. It asks "is this program acting like malware?", watching for actions such as rapidly encrypting many files (ransomware), recording keystrokes (a keylogger), or unexpected outbound connections (exfiltration).

Its properties, and why it is the answer to the unknown:

• Catches never-seen malware. Because it judges by behaviour, not by recognising the code, it can detect brand-new, zero-day malware that has no signature and evades heuristics, as long as the malware does something recognisably malicious, which by definition it must to cause harm. This is the crucial strength, closing the window that signatures leave open.
• Detects by the harmful action itself, which ties directly to the earlier chapters: the spyware architecture's exfiltration stage, ransomware's encryption behaviour, the keylogger's capture, all are behaviours a behavioural detector watches for, which is why understanding malware behaviour is a detection skill.
• Costs and limits. It generally needs to observe the program running (or closely monitor its actions), which is more resource-intensive, and it can also produce false positives when legitimate programs do things that resemble malicious behaviour. And catching malware by its behaviour may mean catching it as it begins to act, so it is paired with the ability to stop and contain quickly.

So behavioural detection is the strongest against the unknown, because harmful malware must act, and acting is what it detects, at the cost of running the program and watching closely, which is exactly what the sandbox (next chapter) provides safely.

The three together

The methods are complementary and used together, which is the practical conclusion:

• Signatures handle the huge volume of known malware cheaply and precisely.
• Heuristics extend to variants and the similar, catching modified known malware.
• Behavioural detection catches the genuinely new by what it does, closing the zero-day gap.

Modern endpoint protection (the chapter after next) combines all three, because none suffices alone: signatures miss the new, heuristics miss the well-disguised, and behavioural detection is costlier and catches malware only as it acts. Layered, they cover known, similar and novel threats, which is the defence in depth the block has taught throughout.

A worked example, framed defensively

A security team evaluates why a novel malware sample was, and was not, caught by their layers, at concept level.

• The sample was brand-new, so signature detection missed it: no signature existed yet, the expected blindness to the unknown.
• It shared some traits with a known family, so heuristic detection flagged it as suspicious, illustrating heuristics catching the similar-to-known, though the team notes heuristics also raised some false positives on legitimate software that week.
• When run in the protected environment, it began encrypting files rapidly, so behavioural detection identified it as ransomware by its actions and it was contained, illustrating behavioural detection catching the novel by what it does, tied to ransomware's known behaviour.
• Conclusion: the layers are complementary, signatures for the known, heuristics for the similar, behavioural for the novel, and the novel sample was ultimately caught by its behaviour, confirming why all three are run together.

The evaluation shows each method's role and limit, and why they are layered. The team detects and analyses; nothing offensive is built.

What beginners get wrong

• Thinking signature detection is enough. It is precise but blind to anything not already in its database, so it misses new and modified malware until a signature exists; the window it leaves open is why other methods are needed.
• Confusing heuristic and behavioural detection. Heuristic inspects what the code looks like (similarity, suspicious traits); behavioural watches what the program does when it runs. Looking-like versus doing is the distinction.
• Believing behavioural detection has no cost. It generally needs to run or closely watch the program and can raise false positives, and it may catch malware only as it starts to act, so it is paired with rapid containment.
• Missing why understanding behaviour aids detection. Behavioural detection watches for exactly the actions the malware chapters described (encryption, keystroke capture, exfiltration), so knowing malware behaviour is a detection skill.
• Treating the methods as competitors. They are complementary and layered; each covers a gap the others leave, which is why endpoint protection uses all three.
• Ignoring false positives. Heuristic and behavioural detection trade some false alarms for catching the unknown; tuning balances detection against disruption.

Quick revision

• Signature: matches a known fingerprint (pattern/hash). Precise, low false positives, but blind to the unknown and needs constant updates; a window exists before a new signature is available.
• Heuristic: flags code that looks like malware (suspicious traits, similarity to known families). Catches variants, at the cost of false positives; still inspects the code.
• Behavioural: watches what a program does at run time and flags malicious actions (encryption, keystroke capture, exfiltration). Catches never-seen/zero-day malware because harmful malware must act; costs: needs to run/watch the program, some false positives, catches it as it acts (so pair with containment).
• Used together: signatures (known), heuristics (similar), behavioural (novel); none suffices alone, so endpoint protection layers all three.

Test yourself

1. How does signature detection work, and what is its fundamental limit?

Signature detection identifies malware by matching it against a database of signatures, distinctive patterns or hashes that uniquely identify known malware samples, so a match near-certainly means that known malware is present, with low false positives. Its fundamental limit is that it can only detect malware whose signature is already in the database, so a brand-new sample, or a modified one whose pattern differs, is missed until a signature is created and distributed, leaving a window during which signature detection is blind to the new threat.

1. How does heuristic detection extend beyond signatures, and what does it cost?

Heuristic detection identifies likely malware by suspicious characteristics or similarity to known families rather than by an exact match, so it can flag modified versions of known malware and new samples sharing malware-like traits that signatures miss, closing part of the blindness to the unknown. It costs certainty: judging by resemblance, it can flag legitimate programs that happen to share suspicious traits, producing false positives that precise signature matching largely avoids, so its sensitivity must be tuned to balance detection against false alarms.

1. What makes behavioural detection able to catch never-seen malware?

Behavioural detection watches what a program does while it runs and flags malicious actions rather than recognising the code, so it does not depend on having seen the malware before. Because any malware must perform some harmful action to cause harm, encrypting files, recording keystrokes, or sending data out, behavioural detection can identify even brand-new, zero-day malware by those actions, closing the window that signature and heuristic detection leave open for threats whose code is unrecognised.

1. Why does understanding malware behaviour, from the earlier chapters, directly support detection?

Because behavioural detection works by watching for exactly the actions those chapters described: ransomware rapidly encrypting many files, a keylogger recording keystrokes, spyware exfiltrating data over the network. Knowing how each malware family behaves tells a defender and a detector which actions signal which threat, so the study of malware behaviour is not academic but is the basis on which behavioural detection recognises malicious programs by what they do, including ones never seen before.

1. Why are the three detection methods used together rather than one being chosen?

Because each has a gap the others cover: signatures are precise but blind to the unknown, heuristics catch the similar-to-known but miss well-disguised novel malware and raise false positives, and behavioural detection catches the genuinely new by its actions but is more costly, may catch malware only as it acts, and also risks false positives. Layering them lets signatures handle the large volume of known malware cheaply, heuristics catch variants, and behavioural detection close the zero-day gap, giving defence in depth across known, similar and novel threats.