Three New Approaches to Command Handling in

NetFPGA Hardware Simulators with Custom

Internal Architecture
Anonymous Author, IEEE Member
Some Technical University

Abstract—Simulations in typical software take a long time.

To make them faster, it is possible to use hardware accelera-
tion. Some people use graphical processors (GPU) which are
faster than typical CPU (Central Processing Unit). Much more
faster and more effective is to prepare Application Specific
Integrated Circuit (ASIC). But it costs a lot of money and also
takes time. Nowadays, optimal solution is to use FPGA chips
(Field-Programmable Gate Array) which allow to prepare pro-
grammable hardware. It is flexible and fast. It requires specific
treatment, tool chain for programming and testing, but finally,
it gives powerful platform for calculation and data processing.
In this paper I will show treatment of new architecture in terms
of command handling. Commands are sent to hardware as IP
packets and after calculation results should be collected. There
are at least 3 ways for this.
Fig. 1. NetFPGA 10G Card

I. I NTRODUCTION
For internal structure of NetFPGA card you can use your
own project or you can modify one of prepared demo projects
which are available on website [1]. In my proposal I focus
on efficient usage of FPGA chip performance. I assumed
new internal structure of code, which allows me to realize
more than one calculation in the same time. In hardware,
you cam dedicate particular part or several ones for some,
very strict, defined tasks. If designer multiply hardware, the
efficiency will be also multiplied. By simple multiplication of
calculation part, we can obtain system of parallel calculating
engines. We have to control them by sending parameters of
tasks and collecting results of calculations. In this paper I will
show three approaches for sending commands and receiving
results, where both are placed in IP packets. I use NetFPGA
card, which can be programmed for serving ethernet traffic.
But, in new architecture, the programmable core will be
changed, only the outer view will be realized as in primary
assumption, i.e. with ethernet frames and IP packets. Due
to strict timing, commands and mainly responses, should
be processed in a proper way. The rest of this paper is
organized as follows: Section 2 gives description of NetFPGA
cards, Section 3 presents usage of NetFPGA cards as a
calculating engine with proposed internal structure, section 4
describes software tools, and Section 5 defines approaches to
sending and receiving commands via IP packets, last section
summarizes paper and signalize future works.
II. N ET FPGA C ARDS
NetFPGA cards (Figure 1) have been developed by Uni-
versity of Cambridge and Stanford University networking
groups as an "open hardware" project [1]–[3]. NetFPGA card
is an extension card for PC, it has four physical network
interfaces (4 × 1Gbps or 4 × 10Gbps electrical or optical
(SF P +) Ethernet ports) and FPGA - programmable hardware
as a main chip. More details about structure, architecture and
usage of such a card can be found in the literature. What is
very important, there is a public framework which implements
basic network functionality (routers, switches and interface
cards) and it is relatively easy to add our own functionality.
In the literature, there is a lot of information about typical
usage and projects prepared by NetFPGA community [4]–[6].
In primary assumption which is presented in Figure 2a,
the main chip of NetFPGA card receives packets from phys-
ical ports (eth0-eth3) as input traffic, analyzes and modifies
them, and sends them out to physical ports as output traffic.
Functionality of main chip is programmable and can be
very flexible. The source and destination of traffic can be
also logical interfaces visible in operating system as nf0-
nf3 interfaces. Very important fact is that this functionality
is realized by hardware, hence, the performance of such an
appliance allows to serve whole traffic with speed of line, in
case of these Ethernet ports it can be 1Gbps or 10Gbps.
 Fig. 2. Schema of modules on NetFPGA card and inside FPGA chip and modified schema of NetFPGA card - modification is only in FPGA chip
Fig. 3. Internal pipeline of modules on NetFPGA card and inside FPGA chip and modified schema of NetFPGA card - modification is only in FPGA chip
III. H ARDWARE PART AS A SIMULATOR
In my apporach, which schematic is presented in Figure
2b, there is no typical modules in FPGA chip. Only commu-
nication with operating system through one interface is used,
it is visible as one active interface in the operating system.
All the traffic sent to this interface is visible by all modules,
they identify traffic dedicated to them by given IP/MAC
addres or even VLAN ID or UDP port. The return path for
data is realized analogically, each module sends traffic with
given parameters. The control application on operating system
serves this interface in promiscuous mode and receives all
frames. Figure 3a presents primary pipeline, where 8 incom-
ing queues and 8 outgoing queues are implemented, because
they are related with serving traffic from two representation of
4 interfaces in two directions. In proposed architecture only
one buffer is realized, because nature of transferred frame is
different. In primary version, 8 incoming traffic streams was
fully served thanks to internal speedup. The main difference
between primary and proposed architecture is to use parallel
custom modules (Figure 3b), where each of them can calculate
its task in the same time. The speedup is not necessary, the
maximal speed of calculation is used by multiple modules in
the same time, hence, the final performance of calculation it
is a product of performance of one module and number of
modules. Proper control and timing mechanisms should be
use to send requests for particular modules and also adequate
way of receiving responses has to be implemented (in module
with name output module arbiter). It reflects on functionality
realized in the software part.
The same FPGA chip (Virtex 5) is used on developers
boards ML555 [7] and ML505 [8] (Figures 4 and 5) which
are available in my laboratory. So the same procedure and
architecture can be used for project for them. On those boards
there are only 1Gbps Ethernet SFP cages, but it is not a
problem, because new architecture is focussed inside fpga
chip, which is the same in all of three analyzed boards.

Fig. 4. ML555 developers board from Xilinx
Fig. 5. ML505 developers board from Xilinx
IV. S OFTWARE PART OF THE SYSTEM
Basic usage of this idea assumes that every custom module
has its own MAC and IP address. All these addresses are in
network which is related with nf0 interface. When operating
system (basing on typical routing table) sends IP packet,
proper module receives ethernet frame with this packet.
Proper configuration of IP/MAC relation in ARP table of host
should be realized, but it is realized automatically by script
during addressing configuration. It is possible to control more
than one card from the same control application (in my labo-
ratory I have 5 NetFPGA 10G cards), proper configuration of
IP addresses, IP networks and routing information has to be
realized. In my case, first assumption was to use Omnetpp
software and prepare software modules related with their
hardware support, i.e. each module (node from Omnetpp)
sends calculation commands for his own calculating module
in FPGA via IP protocol. It was possible but actual version
of my application is not related directly with Omnetpp, but
many functionalities are copied and they are implemented in
my own software (Figure 3). In the Figure 4 there is shown
example of network which can be realized in one fpga chip.
Each switch is represented by one module. In general case,
this system is designated for analysing bigger topologies of
IoT devices.
V. C OMMANDS AND RESULTS IN IP PACKETS
To simulate some system its model has to be prepared.
The calculating part will be realized in hardware, it means
that in VHDL or Verilog modules have to be defined and
implemented in FPGA chip. The software part should also
reflect analyzed system in its internal structures, i.e. model
in hardware has to reflects model in software and vice versa.
This process is complicated and time consuming, it costs a lot
of work. When the simulator is prepared and configured (i.e.
IP/MAC addresses used in software part are also implemented
in custom modules), tasks can be sent to hardware. When
simulator decides to run some task and module for this task
is ready, parameters of tasks are packed into IP payload
and sent to NetFPGA card as typical IP packet packed in
Ethernet frame. This block of data, after passing DMA in
PCI bus, is placed in input buffer and it is visible by every
custom module. There is only one module which has matching
IP/MAC addresses, so, only one custom module will read
this data. The content of IP payload is used as a parameters
for calculations. After finishing calculations, when results are
ready, they are packed into IP packet in order to be received
by DMA and software part. But, the output of custom modules
has to be served in proper way to omit problems. Different
types of problems causes different approach to solving them,
realization of different approaches is described further.
Approach 1: Ask one module and wait
The simples idea assumes that we will send request and
wait for response. One module in FPGA receives tasks, after
their realization it sends out results as IP packet. It will
work, but not efficiently. In the same time only one module
will realize calculations. This approach is relatively easy in
implementation, it can be realized for project, where is no
focus on parallel calculations.
Approach 2: Ask dedicated module and monitor
Second approach assumes that simulator can send requests
for modules before they finish their tasks. They respond with
information about progress (for example "75% done") or final
results if they are ready. This approach is quite good, it
allows to run calculations on multiple modules in the same
time. The software does not wait for end of calculations,
just questioning for actual state of them. In this case control
mechanism are more complicated, but efficiency is very good.
All possible decisions and control of communications is
realized in software. Figure 7 shows example view from
software part of simulator which presents online gathered
results.
Approach 3: Ask dedicated module and wait
We can also imagine situation when module generates IP
packet with results just after ending calculations. It looks
interesting, but it is possible that more than one module will
generate packet in the same time (or almost in the same time).
For proper serving such a cases, dedicated output module
arbiter is necessary. In my approaches, I prefer to use simple
hardware and more sophisticated functionality is realized
rather in software than in hardware for two reasons: 1) It
is easier and faster, 2) and also I can save hardware resources
for bigger number of modules in hardware. Moreover - in
this case, software part has no full control of whole process,
because some decisions are taken in hardware.

Fig. 6. Example of GUI of control application for network simulation
Fig. 7. Example of result view from software when second approach is
realized
VI. C ONCLUSIONS AND F UTURE W ORK
The main chip of NetFPGA card can be programmed.
Although, Designers of NetFPGA cards assumed them to
serve network traffic, it can be used as a almost general
purpose hardware accelerator. When you have a practise with
demo projects for network devices (as switch, card, router),
which use MAC/IP addressing, registers, buffers and PCI
operations, it is easy to implement your own functionality,
which extends the already enormous possibilities even more.
As my future work I plan to use another FPGA chips and
prepare system of tools for convenient and user friendly
simulator. Also automated procedure for huge amount of
IoT devices organized in advanced topologies are considered.
Additional user friendly functionalities will be implemented
and added in software.
ACKNOWLEDGEMENTS
I would also like to thank (in alphabetical order):
• NetFPGA Teams from Stanford University and Uni-
versity of Cambridge [1] for their support and help in
organizing NetFPGA workshop at "my anonymous uni"
and overall work with NetFPGA cards;
• Xilinx University Program [9] for donating to My
University of Technology five NetFPGA cards with 10G
interfaces.
R EFERENCES
[1] Website of community and NetFPGA project: http://www.netfpga.org.
[2] G. Gibb, J.W. Lockwood, J. Naous, P. Hartke, and N. McKeown.
“Netfpga: An Open Platform for Teaching How to Build Gigabit-rate
Network Switches and Routers”, In IEEE Transactions on Education,
Volume: 51, Issue: 3 pp. 364-369, August 2008.
[3] N. Zilberman, Y. Audzevich, G.A. Covington, A.W. Moore: NetFPGA
SUME: Toward 100 Gbps as Research Commodity, IEEE Micro, vol.34,
no. 5, pp. 32-41, Sept.-Oct. 2014, doi:10.1109/MM.2014.61
[4] NetFPGA reference projects: http://www.netfpga.org/project_table.html.
[5]
[6]
[7] Virtex-5 FPGA ML555 Development Kit for PCI and PCI Express
Designs https://docs.xilinx.com/v/u/en-US/ug201
[8] ML505/ML506/ML507 Evaluation Platform User Guide -
https://docs.xilinx.com/v/u/en-US/ug347
[9] Xilinx University Program: http://xilinx.com/support/university.html.