Auditing Lustre file system

Sayed Erfan Arefin

Texas Tech University
Texas, USA
saarefin@ttu.edu

ABSTRACT
With the increasing time, we are facing massive demand for
the increasing amount of data storage. Single-server traditional
architectures are failing to meet the demand of dealing with
the humongous amount of data storage, transfer, and different
events of file systems. Therefore, distributed file systems have
become a necessity in order to deal with the scalability of file
systems. Lustre file system is one of the most popular paral-
lel file systems which is used by most of high-performance
computers nowadays. Lustre auditing plays a vital role and
as a proof of security to support rich security features like
authentication with Kerberos, mandatory access control with
SELinux, isolation, etc. In this project, we will explore luster
auditing using centos in a lustre architecture and represent the
auditing features with a graphical interface.
Author Keywords
Authors’ choice; of terms; separated; by semicolons; include
commas, within terms only; required.
lustre, centos, auditing, parallel file system
INTRODUCTION
We are heading towards network-centric advancement of com-
puter technology where reliable and high-performance storage
systems have become a must. Conventional file system mod-
els with single server architecture tend to fail to achieve the
scalability in accordance to the demand we have. Therefore,
distributed file systems are more common nowadays which
ensures availability, scalability and secured performance.
Lustre filesystem is one of them and it is also one of the
most common parallel and influential file systems which is
used by most of the high performance computers currently.
Lustre is continuously expanding scalability and complexity
which opens the doors for hardware and software vulnerabili-
ties, inconsistent metadata, security threats and administrative
issues[4],[2]. Luster auditing helps to check if there is a possi-
bility of inconsistencies ,failures and administrative errors.
Lustre system gives the user the scope to read, write and persist
data using physical machines as major components which are
Figure 1. Lustre parallel file system
dedicated to some specific roles. It is used to scale file system
performance for massive data. Lustre file systems consist
of three main components: one or more clients, a metadata
service and one or more object storage services shown in figure
1. These three components have their own lustre logic that they
have to run but in case of networking, they converge to the
same point. For example, MDS handles allocation of storage
objects on OSS, OSS provides data storage to clients and
clients use the file system and requests for filesystem events.
In this project, we had a thorough understanding of the lustre
file system and we were able to build it using four different
virtual machines as three different components of lustre. We
showed the lustre auditing information using mongodb.
EXPERIMENTAL SETUP
In this section the description of how we set up the virtual
machines, how we installed the lustre in those vms and the
configurations are given along with how we used changelog
and stored them in mongodb.
Setting up the Virtual Machines
Four virtual machines were used to establish the main compo-
nents of the lustre system, for example, one vm for client, one
vm for MDS and two vms for OSS. We used oracle virtual box
to set up the vms where each vm has 2GB of memory and 20
GB of disk space and Centos version 8.3 as operating system
with kernel version of 4.18.0-240.1.1.el8_x86_64.
On the MDS one Meta Data Target (MDT) was created. A
total number of 6 Object Storage Target (OST) were created
on each of the OSSs. This information can be fetched from
the client by using the “lfs df -h” command. The result after
running this command oon the client cant be seen in Figure 2.
 4. Afterwards, we created a file containing repository defini-
tions for the Lustre packages and stored it in the web server
static content directory which makes it easier to distribute
to the Lustre servers and clients. The commands we used
are as follows:
hn=‘hostname --fqdn‘
cat >/var/www/html/lustre.repo << EOF
[lustre-server]
name=lustre-server
baseurl=https://$hn/repo/lustre-server
Figure 2. All the MDT and OSTs
enabled=0
gpgcheck=0
proxy=_none_
sslverify=0
Lustre Installation
For installing, we copied the lustre packages to an HTTP [lustre-client]
server on the network with the integration into local YUM name=lustre-client
baseurl=https://$hn/repo/lustre-client
repositories. We used the following instructions to establish a enabled=0
web server as a YUM repository host for the Lustre packages. gpgcheck=0
sslverify=0
1. At first we created a temporary YUM repository definition
using following commands which is used to assist the initial [e2fsprogs-wc]
acquisition of Lustre packages. name=e2fsprogs-wc
baseurl=https://$hn/repo/e2fsprogs-wc
enabled=0
cat >/tmp/lustre-repo.conf <<\ EOF
gpgcheck=0
[lustre-server]
sslverify=0
name=lustre-server
baseurl=https://downloads.whamcloud.com/public/lustre
EOF
/lustre-2.14.0/el8.3.2011/server
# exclude=*debuginfo*
gpgcheck=0
5. We installed Lustre Server Software by installing e2fsprogs,
[lustre-client] installing and upgrading the kernel, installing ldiskfs kmod
name=lustre-client
baseurl=https://downloads.whamcloud.com/public/lustre and lustre packages. Next, loaded lustre to the kernel. The
/lustre-2.14.0/el8.3.2011/client commands we used are given below:
# exclude=*debuginfo*
gpgcheck=0 yum --nogpgcheck --disablerepo=*
--enablerepo=e2fsprogs-wc \
[e2fsprogs-wc] install e2fsprogs
name=e2fsprogs-wc yum --nogpgcheck --disablerepo=base,extras,updates \
baseurl=https://downloads.whamcloud.com/public/ --enablerepo=lustre-server install \
e2fsprogs/latest/el8 kernel \
# exclude=*debuginfo* kernel-devel \
gpgcheck=0 kernel-headers \
EOF kernel-tools \
kernel-tools-libs \
kernel-tools-libs-devel
2. Next, we used the reposync command (distributed in the reboot
yum-utils package) to download mirrors of the Lustre repos-
itories to the manager serve using following commands: yum --nogpgcheck
--enablerepo=var_www_html_repo_lustre-server
mkdir -p /var/www/html/repo install \
cd /var/www/html/repo kmod-lustre \
reposync -c /tmp/lustre-repo.conf -n kmod-lustre-osd-ldiskfs \
--repoid=lustre-server --repoid=lustre-client lustre-osd-ldiskfs-mount \
--repoid=e2fsprogs-wc lustre \
modprobe -v lustre
modprobe -v ldiskfs
3. Then, we created the repository metadata with following
instructions:
6. Later, we installed the lustre client and for that we have to
yum install createrepo -y
follow the commands listed in step 5 for upgrading the ker-
cd /var/www/html/repo nel.Then, we installed the kmod package for lustre client us-
for i in e2fsprogs-wc lustre-client lustre-server; do ing following commands and loaded lustre to kernel again:
(cd $i && createrepo .)
done yum --nogpgcheck --enablerepo=lustre-client install \
 kmod-lustre-client \
Lustre-client
modprobe -v lustre
mount -t lustre /dev/vg00/ost1 /mnt/ost1

mount -t lustre /dev/vg00/ost6 /mnt/ost6

Lustre Configuration
The Lustre file system configuration steps should always be
performed in the following order: Client
The client was setup by mounting the lustre mounting point
• Metadata Server using the following commands.
• Object Store Servers 1. First, a mounting point was created using the following
command.
• Client
After the client is configured the lustre file system is usable.
mkdir /mnt/lustre
SELinux and the firewall was disabled in all the virtual ma-
chines. Configuration for all the components are as follows.
Metadata Server 2. The client completed setup by mounting the client mount
The following steps were executed for MDS. point using the following command.
1. The baseline for the MDS is creating a file system named
lustre on the server including the metadata target (–mdt) mount -t lustre 192.168.1.114@tcp0:/lustre /mnt/lustre
and the management server (–mgs).

FILE SYSTEM AUDITING

mkfs.lustre --fsname lustre --mdt --mgs /dev/vg00/mdt
File system auditing is an important section of data organiza-
tion and data integrity. From an organization perspective, file
2. A mount point was created. system auditing enables the system to provide proof that the
data stored in the system is not modified and valid. Certain
criteria of file system auditing are as follows.
mkdir /mdt
• This evaluate the organization’s ability to protect its infor-
mation assets. An organization in this century can have
3. MDS was started using the following command: several terbytes of information regarding its policies, or-
ganization, employee information and data owned by the
mount -t lustre /dev/vg00/mdt /mdt organization. It is the organization’s responsibility to protect
its data from being stolen or corrupted. Thus, file system au-
diting enables the organization to evaluate if the data owned
Object Store Servers and stored for the organization is protected or not.
The following steps were executed for each OSS.
1. Mount points were created for the OSTs; this example uses • It also checks the ability to properly dispense information to
six named ost1 through ost6: authorized parties. An organization can have multiple level
of employees each assigned with different tasks. They have
different sort of access to the organization’s data. Providing
mkdir /mnt/ost1 proper access according to policies to this data is required.
mkdir /mnt/ost6 • Audit consists in controlling that all data accesses made
were done according to the access control policy in place.
2. The command mkfs.lustre command was used to create the • Audit can be used as a proof of security in place.
Lustre file systems.
mkfs.lustre --fsname lustre --ost --index=0
IMPLEMENTATION
--mgsnode=192.168.0.114@tcp0 /dev/vg00/ost1 Lustre has a command "changelog" which can be used to audit
... the file system. Changelog records contain all information
mkfs.lustre --fsname lustre --ost --index=0 necessary for auditing purposes. It also has the ability to
--mgsnode=192.168.0.114@tcp0 /dev/vg00/ost6 identify object of action with file identifiers (FIDs) and name
of targets. This also provides ability to identify subject of
3. The OSS was started by mounting the OSTs to the corre- action with UID/GID and NID information. Every entry
sponding mounting points created in the previous step. The from this command output comes with a timestamp. Which
following commands were used for all the mounting points provides the ability to identify time of action with timestamp.
on all the OSSs.
Prepare for changelog
In order to prepare our implementation to be able to get
changelogs from the MDS virtual machine, the following
commands were executed.
1. It is required to register the user to receive changelogs. To
register a new changelog user for a device ( example: lustre-
MDT0000 ) the following command was executed.
mds# lctl --device lustre-MDT0000 changelog_register
output: lustre-MDT0000: Registered changelog userid
’cl1’
2. It is also required to enable all changelog entry types. In
order to enable all changelog entry types the following
command was executed.
mds# lctl set_param
mdd.lustre-MDT0000.changelog_mask=ALL
output: mdd.seb-MDT0000.changelog_mask=ALL
An example of ta changelog entry of "OPEN" type can be seen
i nthe following example.
7 10OPEN 13:38:55.51801258 2017.07.25 0X242
t=[0x20000401:0x2:0x0] \usepackage{ef=0x7 u=500:500
nid=10.128.11.159@tcp m=-w-}
This command was used ina python program to get changelog
ecery 5 seconds. This change logs are then pushed to a mon-
godb collection. Here the collection is named after the node
name. In this case: MDT-0000. The documents are store in the
mongodb collection. Afterwards the changelogs are cleared in
order to avoid redundancy.
Python 3.8 was used to collect change log information from the
MDS. The output of the command was formatted properly and
stored in the mongodb. A snapshot of the mongodb collection
after running the MDS and executing dummy 100 file creation
of 10 MB files from the client can be seen in figure 3. All the
possible changelog record type can be observed in Table 1 [1].
RELATED WORKS
Dong D et al did a performance study of Lustre file systems.
They focused on the LFSCK operation and found that there
can be cascading errors which may be unrecoverable. This
can be severe problem in case of HPC systems [3].
In the work of Arnab k et al. it can be observed that they
did a study on file system monitoring. There are many tools
available for desktop file system monitoring but there are no
proper tool to monitor the lustre parallel file system. They
proposed a monitoring tool for scalable file system [5].
CONCLUSION AND FUTURE WORKS
In this study lustre installation, configuration and file auditing
functions were implemented. Several queries can be executed
on the data that was collected form the lustre file system and
Figure 3. Documents in mongodb
stored in the mongodb. The future work includes creating an
interactive dashboard with nodejs in order to provide a better
usability of the lustre auditing file system. Another future
work is to detect security vulnerability based on the audit
data collected and stored in mongodb using machine learning
classifiers.
ACKNOWLEDGMENTS
Mr Ahmadian Misha was the mentor for this project.
REFERENCES
[1] 2021. Lustre documentation. (2021).
https://doc.lustre.org/lustre_manual.xhtml
[2] Jinrui Cao, Om Rameshwar Gatla, Mai Zheng, Dong
Dai, Vidya Eswarappa, Yan Mu, and Yong Chen. 2018.
PFault: A General Framework for Analyzing the
Reliability of High-Performance Parallel File Systems.
In Proceedings of the 2018 International Conference on
Supercomputing (ICS ’18). Association for Computing
Machinery, New York, NY, USA, 1–11. DOI:
http://dx.doi.org/10.1145/3205289.3205302
[3] Dong Dai, Om Rameshwar Gatla, and Mai Zheng. 2019.
A Performance Study of Lustre File System Checker:
Bottlenecks and Potentials. In 2019 35th Symposium on
Mass Storage Systems and Technologies (MSST). 7–13.
DOI:http://dx.doi.org/10.1109/MSST.2019.00-20
[4] Lanyue Lu, Andrea C. Arpaci-Dusseau, Remzi H.
Arpaci-Dusseau, and Shan Lu. 2013. A Study of Linux
File System Evolution. In 11th USENIX Conference on
File and Storage Technologies (FAST 13). USENIX
Association, San Jose, CA, 31–44.
https://www.usenix.org/conference/fast13/
technical-sessions/presentation/lu
[5] Arnab K. Paul, Ryan Chard, Kyle Chard, Steven Tuecke,
Ali R. Butt, and Ian Foster. 2019. FSMonitor: Scalable
 Value Description Value Description
MARK Internal recordkeeping LYOUT Layout change
CREAT Regular file creation TRUNC Regular file truncated
MKDIR Directory creation SATTR Attribute change
HLINK Hard link XATTR Extended attribute change (setxattr)
SLINK Soft link HSM HSM specific event
MKNOD Other file creation MTIME MTIME change
UNLNK Regular file removal CTIME CTIME change
RMDIR Directory removal ATIME * ATIME change
RENME Rename, original MIGRT Migration event
RNMTO Rename, final FLRW File Level Replication: file initially written
NOPEN * Denied open RESYNC File Level Replication: file re-synced
CLOSE Close GXATR * Extended attribute access (getxattr)
Table 1. Type of records

File System Monitoring for Arbitrary Storage Systems. Computing (CLUSTER). 1–11. DOI:
In 2019 IEEE International Conference on Cluster http://dx.doi.org/10.1109/CLUSTER.2019.8891045