Efficient Regular Expression Pattern Matching for Network
Intrusion Detection Systems using Modified Word-based
Automata
Pawan Kumar
Supercomputer Education and Research Center
Indian Institute of Science
Bangalore, India
pawan@ssl.serc.iisc.in
ABSTRACT
Network Intrusion Detection Systems (NIDS) intercept the
traffic at an organization’s network periphery to thwart in-
trusion attempts. Signature-based NIDS compares the in-
tercepted packets against its database of known vulnerabil-
ities and malware signatures to detect such cyber attacks.
These signatures are represented using Regular Expressions
(REs) and strings. Regular Expressions, because of their
higher expressive power, are preferred over simple strings to
write these signatures. We present Cascaded Automata Ar-
chitecture to perform memory efficient Regular Expression
pattern matching using existing string matching solutions.
The proposed architecture performs two stage Regular Ex-
pression pattern matching. We replace the substring and
character class components of the Regular Expression with
new symbols. We address the challenges involved in this
approach. We augment the Word-based Automata, obtained
from the re-written Regular Expressions, with counter-based
states and length bound transitions to perform Regular Ex-
pression pattern matching. We evaluated our architecture
on Regular Expressions taken from Snort rulesets. We were
able to reduce the number of automata states between 50%
to 85%. Additionally, we could reduce the number of tran-
sitions by a factor of 3 leading to further reduction in the
memory requirements.
Categories and Subject Descriptors
E.3 [Data]: Data Encryption; C.2.0 [Computer Systems
Organization]: Security and Protection; H.2.7 [Information
Systems]: Security, Integrity, and Protection
General Terms
Cyber Attacks, Intrusion Detection , Memory Footprint,
Performance
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise, to
republish, to post on servers or to redistribute to lists, requires prior specific
permission and/or a fee.
SIN ’12, October 25-27, 2012, Jaipur, Rajasthan, India
Copyright � c 2012 ACM 978-1-4503-1668-2/12/10 ...$15.00.
103
Virendra Singh
Department of Electrical Engineering
Indian Institute of Technology Bombay
Mumbai, India
viren@ee.iitb.ac.in
Keywords
Regular Expressions, DPI, NIDS, Pattern Matching, DFA,
NFA
1. INTRODUCTION
With our increased reliance on the Internet for commerce,
social networking, information acquisition, and information
exchange; intruders have found financial, political, and mil-
itary motives for their actions. Signature/Rule-based NIDS
[1, 18] matches the network traffic against its signature set
and offers good protection against previously known attacks.
However, with growing set of exploits and increasing wire-
speeds, inspecting the network traffic becomes a challenging
task. The emergence of sophisticated cyber attacks have led
to the adoption of regular expressions, over simple strings,
as the language of these signatures [21]. The comparison of
traffic payload against regular expressions, known as Deep
Packet Inspection (DPI), is compute intensive task. The
profiling of NIDS software’s run time estimates the payload
inspection time to be as high as 75% of the total time [8].
Regular expression matching can be performed by con-
structing Nondeterministic Finite Automaton (NFA) or De-
terministic Finite Automaton (DFA). For a given regular
expression with n symbols, an equivalent NFA can be con-
structed with at most 2n states (O(n)). However, NFA en-
gine has polynomial time matching complexity (O(n2 )) per
input symbol consumed. On the other hand, DFAs have
fast matching speed (O(1)) per symbol consumed. However,
DFAs suffer from exponential space complexity (O(2n )) [11].
The NIDS detection should have fast matching speed and
low memory requirements. Hence a DFA like engine is pre-
ferred. The DFA state space blows up exponentially be-
cause of the presence of wild-card (.), character classes ([a-z],
etc.) and kleene star operator (∗) in the regular expressions.
When multiple such expressions are combined into a single
DFA, state space blows up because of replication of smaller
state machines for each wild-card boundary. These patterns
appear frequently in the NIDS signature sets. The state
space blowup problem can be tackled by using two stage
detection [13, 4, 5, 16, 17]. We present a two stage “Cas-
caded Automata Architecture” for efficiently matching such
regular expression signatures. In particular, we make the
following contributions:
• We propose the “Cascaded Automata Architecture” for
a class of regular expressions called Signature Expres-
 sions, that appear frequently in the NIDS signatures.
• We present the challenges involved in two-stage regular
expression matching
• We propose Modified Word-based Automata to ad-
dress these challenges.
• We present efficient implementations of our architec-
ture to help reduce the memory footprint.
The rest of the paper is organized as follows: We present
a survey of recent pattern matching techniques in Section 2.
We formally introduce the “Cascaded Automata Architec-
ture” in Section 3. We present an efficient implementation
and evaluation of our architecture in Section 4. The scope
of future work and conclusion are presented in Section 5.
2. RELATED WORK
2.1 Hardware-based Techniques
Sidhu and Prasanna presented NFA implementation of
regular expressions on FPGA logic components [19]. They
exploited the hardware Flip Flops to store active NFA states.
Brodie et al. [7] presented a pipelined architecture incorpo-
rating run length coding and alphabet encoding compres-
sion techniques to achieve single-cycle transitions. Divyas-
ree et al. [9] proposed an NFA based implementation of
regular expressions with “extended1 ” regular expression syn-
tax. Kaneta et al. [12] presented “Dynamic reconfigurable
bit-parallel NFA architecture” for a sub-class2 of regular ex-
pressions. They encode the given NFA in “bit-masks” that
are stored in a collection of registers and block RAM. Naka-
hara et al. [17] proposed regular expression matching cir-
cuit based on a decomposed “Modular NFA with Unbounded
string transition” obtained from an NFA by replacing a series
of states with a single transition string.
2.2 Generic Compression Techniques
Yu et al. [24] presented regular expression re-writing and
grouping techniques to reduce the memory usage. Kumar
et al. [14] introduced the terms “insomnia”, “amnesia”, and
“acalculia” to describe the deficiencies of traditional DFA.
They augmented the DFA with history buffer and counters
to address these limitations. Smith et al. [20] target the
state-space blowup problem of the DFAs. The paper intro-
duces XFA which augment the Finite Automata (FA) with
finite scratch memory and instruction to manipulate this
memory. Ficara et al. [10] introduced δFA where they store
only the differences between adjacent states. Wang et al.
[23] proposed a length-based matching scheme which con-
verts the original byte-stream into a shorter integer stream
and then matches it with a variant of DFA, called StriD2 FA.
Kumar et al. [15] introduced D 2 F A that is constructed
from a DFA via incrementally replacing several of its transi-
tions with a single default transition. Zhang et al. [25] pro-
posed a regular expression matching algorithm using tran-
sition merging, called TM-DFA. Their algorithm is based
on the observation that in most cases the adjacent states
have common destination states when consuming the same
character.
1
Including counting constraints
2
same as the class targeted by our architecture
104
Baker et al. [4] proposed the use of “software-decelerated”
approach, where they utilize a small custom microcontroller
linked to each regular expression matching engine to provide
complex pattern matching functionality. Becchi and Crow-
ley [6] presented a hybrid-FA (H-FA) based approach target-
ing the state-space explosion problem. Lin and Hsiao [16]
proposed a “Hierarchical state machine” using “master” and
“slave” state machines. Their “master” state machine identi-
fies the substrings of regular expressions and the “slave” state
machine keeps track of the regular expression. Bando et al.
[5] proposed Look ahead FA which performs out-of-sequence
detection of different components of regular expression, viz.
“substrings”, “character classes”, and “bounded/unbounded
repetitions” using specialized detectors. Tang et al. [22] ad-
dressed the DFA state explosion. They proposed “fcq-FA”
(flag/counter and queue FA) which uses pipelined queues
and counters for remembering matching history.
3. CASCADED AUTOMATA ARCHITECTURE
3.1 Key idea
Consider few regular expressions taken from Snort rule
set:
%3f\s+url=[^\n\r]*httpUser-Agent\x3A (1a)
e2give\.com.*Host\x3A.*Host\x3A (1b)
e2give\.com.*Redirector\x3A.*Host\x3A (1c)
FILESIZE\x3E[^\n\r]*\x3E\d+ (1d)
We observe that a significant number of regular expres-
sions appearing in the NIDS signatures are of the form sub1
β op sub2 β op . . . (∈ SigRE); where subi is a substring, β(⊆
Σ, alphabet) is a character class (CC) and op ∈ {?, ∗, +}. We
consider only such regular expressions in our design. Many
of these substring components are common, or share pre-
fixes. Matching these substrings can be optimally imple-
mented using existing string matching algorithms. We sub-
stitute these substrings and character classes by a new sym-
bol and perform pattern matching using these new symbols.
The presence of wild-cards with repetitions (e.g. .∗, .+, etc.)
causes the states corresponding to substrings to be repli-
cated leading to state-space explosion. By replacing the sub-
string components (subi ) with new symbols (Si ), we could
avoid state-space explosion and reduce the memory foot-
print. Replacing character classes (βi ) with a single symbol
(Ci ) also helps reduce the total number of state transitions
in the new automata.
3.2 Overview
In “Cascaded Automata Architecture”, we propose to com-
pile substring (subi ) and character class (β) components into
first stage automata. This automata reads a character from
the input stream and generates a symbol on every clock cy-
cle. Each substring or character class, when matched at the
first stage, generates a unique symbol for the second stage
automata. A special symbol ‘0’ is generated if the first stage
automata doesn’t find any match. This stage can be imple-
mented as an Aho-Corasick state machine [3] or a DFA. We
focus on the design and implementation of the second stage
automata. Figure 1 presents our architecture.
 ���������������������� ����������� ������������
������������������ ��� ������� ����� ���������� state ‘0’ is the start state, and double circled states repre-
�� ���������������� ������������������������
sent matching state with the label ‘Ri ’ indicating matched
����������� �
�
�� �������� regular expression. The label of a state contains the state
� � �����������
�� � �� �� ����
number and its depth from the start state, separated by ‘-’.
� � � � �

3.3 Challenges
��
� � ��
�

Replacing substring (subi ) and character class (βi ) com-

�� �� �� �� ������� ����� ponents with new symbols Si and Ci , respectively is not
�������������������� ������������ �������� straight-forward. The regular DFA’s states store the infor-
mation about partial matches of different regular expressions
Figure 1: Cascaded Automata Architecture at any point of time. This information is not present in the
W-NFA. In this Section, we present the challenges in cas-
caded automata approach.
The second stage automata consumes the symbol gen-
erated from the first stage. In order to maintain equiva- 3.3.1 Timing Information
lence with the original NFA, transitions corresponding to The state of W-NFA is not capable of storing timing in-
substrings are stored with length information. This length formation. For example, the state number ‘5’ of Figure 2
information is stored along with a counter array in a small doesn’t contain any information about when to expect the
global table that can validate a substring based transition to symbol ‘S2 ’ (corresponding to substring “url=”). For this
another state. This global table counter is updated when- W-NFA, state ‘5’ is activated upon receiving symbol ‘C1 ’
ever a state with length-bound transition is entered or left. (corresponding to character class “\s”). Hence, symbol ‘S2 ’
is expected exactly after three cycles (i.e. on fourth cycle) of
ORIGINAL RE REWRITTEN RE
state ‘5’ being activated. We need to associate this “timing
%3f\s+url=[^\n\r]*httpUser-Agent\x3A S1 C1 + S2 C2 ∗ S 3
e2give\.com.*Host\x3A.*Host\x3A S4 C3 ∗ S 5 C3 ∗ S 5
information” with certain4 states as and when required.
e2give\.com.*Redirector\x3A.*Host\x3A S4 C3 ∗ S 6 C3 ∗ S 5
FILESIZE\x3E[\^\n\r]*\x3E\d+ S7 C2 ∗ S 8 C4 + 3.3.2 Multiple Instances
It is easy to observe that for the state ‘5’, only one instance
Table 1: Re-written Regular expressions
can remain active at any instance because it is activated by
receiving symbol ‘C1 ’ from state ‘1’. Upon receiving symbol
‘C1 ’, the already active instances of the state ‘5’ can not
remain active any longer (because ‘S2 ’ doesn’t contain any
�����
����
�
symbol belonging to class ‘C1 ’). However, there will be sit-
���
uations when a state can be entered multiple times. Hence,
we need to keep track of multiple instances of an active state.
��� ���������� ������������ ����������
This is important because “timing information” is associated
with each active instance of a state.
��� �� ��� ���
� ��� ������� ��� ���

3.3.3 Overlapping Issue

�� �������������� ���� �������� Consider a regular expression “abc. ∗ bcd”. The W-NFA
corresponding to this expression is shown in Figure 3. For
��� ��� ���
� ��� �� ��� ���
� this W-NFA, state ‘1’ is activated from state ‘0’ upon re-
ceiving “abc”. However, if the input stream is “abcd”, the
���� �������� �� �������� first stage will generate symbol corresponding to “abc” upon
reading first three characters. Upon reading next character,
���� ���� ���� it will generate symbol corresponding to “bcd”. However,
��� �������
�� �� �� state ‘1’ has no information to distinguish this symbol gen-
erated by using suffix of previously generated symbol. This
������������������ will lead to false match. We need to augment W-NFA with
overlapping information to avoid false match.
����
��

� �
Figure 2: Word-based NFA for re-written RE
����� ��� ��� ���
���
���� ��

The state machine operating upon the rewritten regular

expression of Table 1 can be obtained by generating the
NFA of re-written regular expressions. We term this ma- Figure 3: W-NFA illustrating the overlapping issue
chine as Word3 -based NFA (W-NFA). W-NFA for regular
expressions of Table 1 is shown in Figure 2. In Figure 2,
3 4
here, Word refers to substring component not all states require this timing information

105
3.3.4 Overlapping Component Symbol Generation
Some character classes might partially overlap with each
other leading to the issue of simultaneous generation of mul-
tiple symbols. For example, for character classes β1 = {a, b, c}
and β2 = {c, d, e}, both character classes will be simulta-
neously active for β1 ∩ β2 = {c}. Similarly, some character
classes will be be superset of others. In addition, sometimes,
both, a character class as well as some substring, might be
active simultaneously. For example, substring ‘home’ and
β2 will be active simultaneously upon receiving the last al-
phabet of ‘home’. We need to be able to efficiently encode
such information to be passed on to second stage automata.
Our aim is to generate only one symbol (substring or
character class) per clock cycle from the first stage
to the second stage. This ensures performance against
maliciously crafted flows.
In the next Section, we introduce W-NFA. We augment
the W-NFA with required information to address the chal-
lenges in cascaded automata architecture.
3.4 W-NFA and Proposed Enhancements
Definition 1. Modified Alphabet (Σm ): A finite set of
symbols representing substring (subi ) and character class
components (βi ) of the signature set S
Definition 2. Substring Set SST R : A finite set of all the
substring components (subi ) present in the signature set S
Definition 3. A Word-based NFA is a quintuple
M = (Q, Σm , Δ, s, F ), where
• Q is a finite set of states
• Σm is modified alphabet
• s ∈ Q is the initial state
• F ⊆ Q is the set of final states, and
• Δ : Q × Σm → 2Q , the transition function
Example: Figure 2 and 3
A Word-based NFA corresponding to signature set S is the
combined �-free NFA obtained from the elements of S, with
modified alphabet Σm , using Thompson’s NFA construction
algorithm.
3.4.1 Language of a state
The substring components and character classes may not
always be independent. While a symbol for substring or
character class is generated, another substring match might
be in progress. For example, for strings “root” and “other”,
a match for “root” means a partial match for “other”. Thus
the state which becomes active after receiving symbol cor-
responding to “root” partially matches the symbol “other”.
Hence, in that state, we may receive symbol correspond-
ing to “other” ‘3’ clock cycles after activation of that state.
This information about partial matches need to be associ-
ated with W-NFA states to validate transitions. We term
the set of all possible partial matches (i.e. strings that are
prefix of some member of SST R ) that are reachable at a
state as “language” of that state. Formally, we define two
type of languages associated with the states.
106
Definition 4. Incoming language LI of a state q ∈ Q
of W-NFA M is defined as the set of all substrings, that
are proper prefix of some element of SST R , that might be
matched in the first-stage string matching automata when
the state q is just5 activated.
Definition 5. Partial language LP of a state q ∈ Q
of W-NFA M is defined as the set of all substrings, that
are proper prefix of some element of SST R , that might be
matched in the first-stage string matching automata while
the state q is active
The states in the W-NFA are traversed after receiving
“word” symbols. These states, however, contain implicit in-
formation about the possible partial matches at that state.
The incoming language of a state determines the set of par-
tial matches that can be active in the first stage automata
when this state is activated in the second stage automata.
The partial language of a state determines the set of par-
tial matches that can be active in the first stage automata
while this state is active in the second state automata. The
incoming language of starting state is empty set ‘∅’. The in-
coming language of other states can be obtained recursively
from the partial language of their preceding6 states. Algo-
rithm 1 shows our pseudo-code to compute LI and LP for
W-NFA states.
Algorithm 1 Pseudo-code to compute LI and LP
1: procedure ComputeLanguage(M ) � M : Input W-NFA
2: for all q ∈ M.Q do
3: M.q.LI ← ∅ � Initialize the incoming language
4: Queue ← Breadth first traversal of W-NFA states M.Q
5: while Queue.N otEmpty() do
6: Current ← Queue.F irst() � Pop front element
7: Current.LP ← Substrings of str ∈ SST R , generated
8: using Current.self loop transitions and Current.LI
9: for all Symbol ∈ Σm do
10: for all T arget ∈ M.Δ(Current, Symbol) do
11: T arget.LI ∪ ← Substrings of str ∈ SST R that
12: can be generated using Current.LP and Symbol
3.4.2 Overlapping Component Symbol Generation
The SigRE components may have some relation with each
other. For example, S1 = “terServer”, S2 = “Server”, and
β1 = {a, b, . . . , z} are interrelated. A generation of sym-
bol S1 guarantee the occurrence of symbols S2 and β1 , but
not vice-versa. Similarly, the character classes may have
non-empty intersection with each other. However, we don’t
want to generate multiple symbols at any stage. Therefore,
we provide a solution to efficiently map this information to
modified alphabet set Σm in a lossless way.
Substring overlapping issue:
If the detection of a substring component Si implies the
presence of some smaller substring Sj or character class com-
ponent βk , only symbol corresponding to Si is produced. To
maintain the equivalence with the original state ma-
chine, we pre-compute the additional possible transitions
(with label Si ) for each state along with possibly existing
transitions with label Sj and βk . These newly computed
transitions ensure that those state traversals take place with
Si , in case there is a transition with label Sj or βk . Pseudo-
code to compute this information is presented in Algorithm
2. Consider the regular expression set:
5
the state is not reactivated using a self-loop from itself
6
states that have an outgoing transition to this state
 ���� ���� ����� ������
���
���������� ��� �������� ���
abc[\d]def (2a)
����
����
kabc[a-z]*bcdef (2b) ����

���� ���� ����� ������

��� ���
Figure 4 shows the W-NFA with additional transitions for �������� ���

regular expression set 2. �� ���

��� ��� �����

Algorithm 2 Pseudo-code to compute additional transi- ������

tions for W-NFA states ����� �������

1: procedure AdditionalTransitions(M ) � M : Input W-NFA ��� ���

2: Queue ← Breadth first traversal of W-NFA states M.Q �� ��
3: while Queue.N otEmpty() do
4: Current ← Queue.F irst()
5: Δ� ∪ ← {(Current, Sstr ), Current}, where Sstr are
6: symbols for str ∈ SST R that can be generated using
7: Current.self loop transitions and Current.LP Figure 5: W-NFA with cb-state and lb-transitions
8: for all Symbol ∈ Σm do
9: for all T arget ∈ M.Δ(Current, Symbol) do
10: Δ� ∪ ← {(Current, Sstr ), T arget}, where Sstr are • βi ∩ βj �= ∅ and βi �⊆ βj and βj �⊆ βi , i.e. some symbol
11: symbols for str ∈ SST R that can be generated
12: using Current.LP and Symbol will generate only Ci , some only Cj , some both Ci and
13: M.Δ∪ ← Δ� � Update the transitions Cj , while remaining will generate neither Ci nor Cj

For the first case, there is no issue. For the second case,
we can partition βj into βi and βj − βi . For the last case, we
need to generate three character class symbols corresponding
to character classes, βi − βj , βj − βi , and βi ∩ βj respectively.
���
���� ���� ����� ������ Then, we need to modify the transition information associ-
����� ������ ����� ����
ated with states. For example, for the second case, each
����
���� transition with label βj is replaced with a transition with
����
the same start and destination states but new labels βi and
��� ���
���� ���� ����� ������
�����
β j − βi .
This approach can be extended for the p character class
���� �����
components, β1 , . . . , βp , of the signature expression set S.
���
���
�� 3.4.3 Length-bound Transitions
���
The ‘length-bound’ (lb) transitions compliment the ‘counter-
based’ states to augment ‘timing information’ with them.
��� The length information associated with such transitions,
��
along with the qualifying label, is used to determine if this
transition is taken.

Figure 4: W-NFA with pre-computed additional struct length_bound_transition {

transitions
Character class overlapping issue:
After consuming an input symbol, the first stage automata
check it for the possible generation of symbol for character
class components. If the input symbol belong to multiple
character classes, multiple character class components will
be generated simultaneously. However, we want to generate
only one symbol per cycle. Therefore, we present a solution
to form new character classes (and corresponding symbols)
so that a symbol can match utmost one character class.
For any pair of new character class symbols Ci and Cj ,
their corresponding character classes, βi and βj , will be re-
lated in one of the following three ways:
• βi ∩ βj = ∅, i.e. either none7 of them, or only one of
symbol Ci and Cj can occur
• βi ⊆ βj , i.e. occurrence of symbol Ci implies occur-
rence of Cj
7
because input symbol may not belong to any of βi or βj
symbol label;
state *target;
int length; // Timing Info
};
The normal word transition is represented as a transition
with ‘label’ tag in the W-NFA. We represent the length-
bound transition with ‘length#label’ tag in the W-NFA graph.
3.4.4 Counter-based States
States that require “timing information” about when to
expect word transitions can be associated with a “counter
array”. The number of elements in this “counter array” is
equal to the number of maximum possible active instances
of a state. Each “active instance” of the counter corre-
sponds to an active instance of the state. Upon receiving
the corresponding word symbol, the length information as-
sociated with such word labels is matched against the active
instances of “counter array”. A transition is made iff the
length matches with the count of one of the counter of the
“counter array”. The length of word labels determines the

107
maximum count of the corresponding counter before deac-
tivating it. A counter-based state has only length-bound
transitions.
For example, state ‘5’ of Figure 2 can be associated with
a single counter. Since, at any time, only one instance of
this state can be active, only a single counter is sufficient
to accommodate the “timing information”. Since the word
transition out of this state is “4” characters long, the counter
can be deactivated if symbol ‘S2 ’ is not received at the 4th
cycle. A “counter array” can also be deactivated if some
symbol is received that leads to automatic expiry of this
state. Figure 5 represents the W-NFA, with cb-state and lb-
transitions, corresponding to regular expression set 2. The
counter arrays have associated operations to activate, de-
activate, remove, and update an instance. Additionally,
they support enquiry function to validate a lb-transition.
Definition 6. A counter-based (cb) state, qcb ∈ Qcb , has
an associated “counter array” to maintain the “timing in-
formation”. All transitions associated with such states are
length-bound transitions.
Counter-based State for Overlapping Issue:
As seen in Section 3.3.3, the “overlapping issue” occurs
if at any state, there exists a transition that can be gener-
ated using partial substring from a transition that leads to
this state. Because of the “overlapping issue”, there might
be a false match if this transition is generated using par-
tial substring which was already received upon entering this
state. Such a transition should not be acted upon. However,
without augmenting “timing information”, it is not possi-
ble to differentiate between the transition generated using
partially matched substring from previous state, and the
transition generated upon obtaining the entire string after
reaching the current state. Formally,
Definition 7. The Overlapping issue is said to occur at
a state q, if ∃str ∈ LI (q) such that str is prefix of some
word-transition out of state q.
Definition 8. The Front Symbol of a (non-null) string
‘str’ is its leftmost symbol
FrontSymbol(a0 a1 . . . an ) = a0
Due to its nature, “overlapping issue” exists only for word-
based transitions. This problem can be solved by introduc-
ing a new state between the state that suffers from over-
lapping issue and the target state of the word-based tran-
sition that caused overlapping issue. The new state, just
introduced, is reached from this state upon receiving Front
Symbol of the word-based transition that caused overlap-
ping issue.
Counter-based State for Timing Information:
If there is no “overlapping issue” at a particular state, we
may still need to augment timing information. If a state has
only word-based transitions, we need to augment “timing
information” to it. Similarly, if a state has self-loop transi-
tions on character class symbol, we will need to check if it is
required to add timing information, or not. For example, if
a state with no-overlapping issue has transitions on ‘Sbcdef ’
and ‘C[a−z] ’, there is no need to add timing information to
this state. Its reason is that while a match for ‘Sbcdef ’ is
in progress, the symbol ‘C[a−z] ’ will be generated every cy-
cle and keep this state active. On the contrary, consider
108
the state with transitions on ‘Sab11cd ’ and ‘C[a−z] ’. Here,
upon receiving the partial match ‘ab1’, the self-loop symbol,
‘C[a−z] ’ will not be generated8 . Hence, we need to have tim-
ing information after this point to perform exact matching.
3.5 Modified Word-based NFA
So far we have seen the challenges faced in using two phase
detection engine. We introduced the counter-based states,
counter-arrays, length-bound transitions, and precomputed
languages associated with states, to solve these issues. Now,
we present a formal model of our Modified W-NFA.
Definition 9. A Modified W-NFA (M-WNFA) obtained
from the W-NFA M is a sextuple Mm = (Qm , Σm , CA, Δm , s, F ),
where
• Qm (= Q ∪ Qcb ) is the set of W-NFA states along with
counter-based states introduced to augment the W-
NFA with timing information (using counter-arrays)
• Σm is set of modified symbols (Definition 1)
• CA is the set of counter-array (2N ) associated with
counter-based states
• s ∈ Qm is the initial state
• F ⊆ Qm is the set of final states, and
length N
• Δm : (Qcb × 2N ) × Σm −−−−→ 2(Qcb ×2 )
Definition 10. The configuration of a state machine with
counter-based state is defined as (qcb , ca(qcb )) ∈ Qcb × 2N .
The transition is taken iff the state configuration has an
entry corresponding to “length”. The transition function up-
dates the configuration of a state machine’s source state as
well as destination state. The updation of the source state
configuration is required as it includes updating the timing
information stored in the counter-array. The destination
state, similarly needs to be updated with timing informa-
tion, such as adding an instance of active state, updating
the current active state counters, etc. The normal states
(without any associated counter-array) may be considered as
a special class of counter-based states with empty counter-
array. The transitions without any associated length infor-
mation, may similarly be considered as the special case of
lb-transitions that can match any length.
Example: Figure 6 presents M-WNFA for regular expres-
sion set 2. State ‘q2 ’ has associated counter-array ‘ca(q2 )’.
State ‘q5 ’ is another counter-based state with counter-array
‘ca(q5 )’. The action associated with transitions related to
counter-based states has been highlighted along with the
transitions. In addition, the counter-arrays are updated ev-
ery cycle. Also, if a counter-based state receives a symbol
(other than character class) not shown in the state graph,
the counter-array corresponding to that state is deactivated.
4. EVALUATION
The second-stage automata of our architecture is modeled
as M-WNFA. However, for fast matching speed, we convert
M-WNFA to its deterministic counterpart, Modified Word-
based DFA, (M-WDFA) using slight modification in subset
8
1 �∈ [a − z]
 ��
����� ����� ������ ������� RE DFA M- % Red- new cb- counter
����������� ���� ��������� ����
num States WDFA uction symbol State array
����� States in states num num (bits)
�����
�����
37 32512 4814 85 81 20 126
����� ����� ������ �������
�� �� ��������� ���� 16 79210 37776 53 95 59 257
���������������� ����
40 98151 20022 80 86 27 224
��� ����������������
22 22230 10952 51 55 27 274
�������
������ �� �� ������
������ Table 2: DFA and M-WDFA memory requirements
����������������� ������ ��������
�����������������
for different regular expression sets
�� ��
�� �� Signature DFA M-WDFA H-FA D2 F A
Set (#RE) (MB) (MB) (MB) (MB)
Snort1 (37) 15.9 2.3 10.6 0.9
Figure 6: M-WNFA corresponding to regular ex- 85.6% 33.4% 94.3%
pression set 2 Snort2 (16) 77.4 8.0 10.8 6.4
89.7% 86.0% 91.7%
construction algorithm to take care of cb-states. The transi- Snort3 (40) 95.8 15.9 19.7 7.0
tion, in this M-WDFA depends upon both the input symbol 83.4% 79.4% 92.7%
as well the qualifing values from the counter-array associ- Snort4 (22) 21.7 7.5 8.0 1.8
ated with cb-states. Those states, whose subset consisted of 65.5% 63.1% 91.7%
cb-states, needs pointers to corresponding counter arrays.
Table 3: Memory requirements of DFA, M-WDFA,
Definition 11. A M-WDFA obtained from the M-WNFA H-FA, and D 2 F A for different regular expression sets
Mm is a sextuple MD = (QD , Σm , CAD , ΔD , s, F ), where
• QD is the set of M-WDFA states along with associated The main memory compression comes from the reduction
counter-arrays corresponding to the counter-based M- of number of states in the proposed M-WDFA. Also, the
WNFA states associated with them reduction of the alphabet size reduces the memory required
to store the state transition table by a factor of about 3.
• Σm is set of modified symbols (Definition 1) The overhead is the pointers that each state need to link to
• CAD is the set of counter-array (2N ) associated with the counter-based states. Our experiments showed that any
counter-based states of Mm MWDFA state had at most two counter-arrays associated
to them. One counter array can be addressed using 6 bits
• s ∈ QD is the initial state (assuming maximum number of counter-based state to be
59). Hence a total of 12 bits need to be associated with each
• F ⊆ QD is the set of final states, and state. Also, the counter-array can be implemented as bit
• ΔD : QD × Σm × CAD → QD × CAD , the transition vector. The total number of bits required to implement such
function arrays are very small. The table doesn’t show the memory
requirement of the f irst stage automata because it is linear
Counter array implementation as bit vectors: in the size of the signature set and is quite small compared
The counter-arrays associated with states can implemented to the size of DFA and M-WDFA.
as bit-vectors of width equal to the maximum possible counter Table 3 shows that the memory compression obtained
value. The value of an instance is associated with the bit- from M-WDFA is superior to that from H-FA. The com-
position and the effect of incrementing the counter-arrays is pression obtained is lesser than D 2 F A construction. This is
achieved using left shift operator on the bit-vector. The ac- expected as D 2 F A is a graph based compression technique
tivation of a state instance corresponds to setting the right- where compression in the DFA graph is obtained by shar-
most bit of the bit-vector. The instance expires when it ing transitions between different states using default transi-
shifts out of the bit-vector from the leftmost position. tions. However, M-WDFA has very small run time overhead
We implemented our architecture using the “Regular Ex- compared to D 2 F A. Table 4 shows the run time results of
pression Processor [2]”.We modified the tool to generate the M-WDFA compared to standard DFA and D 2 F A. The run
M-WDFA. Also, we implemented the f irst stage of the ar- time is obtained by parsing traces of 4MB size. The traces
chitecture using Aho-Corasick algorithm as our aim was to consisted of roughly 5% malicious content. The system used
consider the memory requirements. We evaluated our ar- for the test is Intel Core 2 Duo, 2.50 GHz processor with
chitecture on regular expressions extracted from the recent 4GB RAM. The run time results show that DFA performs
(mid 2011) Snort ruleset. In the experiments which we fol- the fastest matching. D 2 F A requires 5 to 6 times more time
low, the expressions has been grouped so as to obtain DFA compared to DFA on a general purpose PC. Whereas M-
with reasonable size. The dataset consisted of regular ex- WDFA has a matching overhead of roughly 12% compared
pressions of the type SigRE. Some of the expressions were to standard DFA. The overhead is because of updation and
slightly rewritten to make them of the class SigRE. A repre- enquiry on Counter-Arrays associated with counter based
sentative of different types of regular expressions have been states.
taken. We compare our design with the baseline DFA re-
sults. Table 2 presents the results. 4.1 Comparison with Related Work

109
 Signature RE DFA M-WDFA D2 F A
Set num (ms) (ms) (ms)
Snort 1 37 60 64 299
Snort 2 16 50 56 288
Snort 3 40 66 74 319
Snort 4 22 50 54 330
Table 4: Run time results of DFA, M-WDFA, and
D 2 F A for different regular expression sets
Lin and Hsiao [16] proposed to divide the state machine
into “master” and “slave” components. However, they tar-
geted the regular expressions of the form “.∗S1 .∗S1� ”, “.∗S2 .∗
S2� ” (where Sk represents a string), etc. They also identify
the exception problem, where the substrings overlapping
within the same expression (i.e. the suffix of S1 is a prefix
of S1� ) cause false match. However, they don’t provide any
solution for the exception problem. We discussed all the is-
sues involved in using a two level design. Also, we considered
broader class of regular expressions.
Other proposals like [4, 5, 22, 12] rely on vast hardware
resources present on the FPGAs, whereas we propose a DFA
based solution, suitable on both FPGAs as well as General
Purpose Computers. Our architecture is orthogonal to some
proposals like [15, 25, 20, 10].
5. FUTURE WORK AND CONCLUSION
Our architecture doesn’t consider regular expressions with
counting constraints. However, like other proposals [14, 20],
we also use counters for tracking information. Extending
our work for a wider class of regular expressions present an
interesting future direction. Also, with some modifications,
the first-stage of our design can be used to quickly filter out
traffic to decrease the traffic load for the second stage.
We have presented “Cascaded Automata Architecture” that
can efficiently utilize the existing research done on string
matching problem for NIDS applications. We addressed the
issues related to using string matching solutions for regu-
lar expression pattern matching and proposed their solution
with our “Modified Word-based Automata”. Our prelimi-
nary evaluation shows significant memory savings. With ef-
ficient FPGA implementation of our architecture, we expect
to be able to accelerate the NIDS detection engines.
6. REFERENCES
[1] http://www.snort.org/.
[2] http://regex.wustl.edu/index.php/Main_Page.
[3] A. V. Aho and M. J. Corasick. Efficient string
matching: An aid to bibliographic search.
Communications of the ACM, June 1975.
[4] Z. K. Baker, H.-J. Jung, and V. K. Prasanna. Regular
Expression Software Deceleration for Intrusion
Detection Systems. FPL, 2006.
[5] M. Bando, N. S. Artan, and H. J. Chao. LaFA:
Lookahead Finite Automata for Scalable Regular
Expression Detection. ACM/IEEE ANCS, 2009.
[6] M. Becchi and P. Crowley. A Hybrid Finite
Automaton for Practical Deep Packet Inspection.
ACM CoNEXT Conference, December 2007.
[7] B. Brodie, R. Cytron, and D. Taylor. A Scalable
Architecture for High-Throughput Regular-Expression
110
Pattern Matching. ISCA, 2006.
[8] J. Cabrera, J. Gosar, W. Lee, and R. K. Mehra. On
the Statistical Distribution of Processing Times in
Network Intrusion Detection. IEEE Conference on
Decision and Control, December 2004.
[9] J. Divyasree, H. Rajashekar, and K. Varghese.
Dynamically Reconfigurable Regular Expression
Matching Architecture. Application-Specific Systems,
Architectures and Processors, 2008.
[10] D. Ficara, S. Giordano, G. Procissi, F. Vitucci,
G. Antichi, and A. D. Pietro. An Improved DFA for
Fast Regular Expression Matching. SIGCOMM, 2008.
[11] J. E. Hopcroft, R. Motwani, and J. D. Ullman.
Introduction to Automata Theory, Languages, and
Computation. Addison-Wesley, 2000.
[12] Y. Kaneta, S. Yoshizawa, S. ichi Minato, H. Arimura,
and Y. Miyanaga. Dynamic Reconfigurable
Bit-Parallel Architecture for Large-Scale Regular
Expression Matching. FPT, 2010.
[13] P. Kumar and V. Singh. Efficient Regular Expression
Pattern Matching using Cascaded Automata
Architecture for Network Intrusion Detection System.
9th IEEE EWDTS, September 2011.
[14] S. Kumar, B. Chandrasekaran, J. Turner, and
G. Varghese. Curing Regular Expressions Matching
Algorithms from Insomnia, Amnesia, and Acalculia.
ACM/IEEE ANCS, 2007.
[15] S. Kumar, S. Dharmapurikar, F. Yu, P. Crowley, and
J. Turner. Algorithms to Accelerate Multiple Regular
Expressions Matching for Deep Packet Inspection.
SIGCOMM, October 2006.
[16] C.-H. Lin and H.-S. Hsiao. Hierarchical State Machine
Architecture for Regular Expression Pattern
Matching. ACM GLSVLSI, 2009.
[17] H. Nakahara, T. Sasao, and M. Matsuura. A Regular
Expression Matching Circuit Based on a Decomposed
Automaton. Applied Reconfigurable Computing, 2011.
[18] V. Paxson. Bro: A System for Detecting Network
Intruders in Real-Time. In Computer Networks,
31:2435–2463, 1999. http://bro-ids.org/.
[19] R. Sidhu and V. K. Prasanna. Fast Regular
Expression Matching Using FPGAs. FCCM, 2001.
[20] R. Smith, C. Estan, and S. Jha. XFA: Faster
Signature Matching with Extended Automata. IEEE
Symposium on Security and Privacy, May 2008.
[21] R. Sommer and V. Paxson. Enhancing Byte-level
Network Intrusion Detection Signatures with Context.
ACM CCS, 2003.
[22] Y. Tang, J. Jiang, C. Hu, and B. Liu. Managing DFA
History with Queue for Deflation DFA. Information
Security Conference, January 2011.
[23] X. Wang, J. Jiang, Y. Tang, B. Liu, and X. Wang.
StriD 2 FA: Scalable Regular Expression Matching for
Deep Packet Inspection. IEEE ICC, 2011.
[24] F. Yu, Z. Chen, Y. Diao, T. V. Lakshman, and R. H.
Katz. Fast and Memory-Efficient Regular Expression
Matching for Deep Packet Inspection. ANCS, 2006.
[25] J. Zhang, D. Zhang, and K. Huang. A Regular
Expression Matching Algorithm using Transition
Merging. IEEE Pacific Rim International Symposium
on Dependable Computing, 2009.