Executive summary: The Splunk software’s configuration file is currently open to editing or deletion

by any users, which caused an issue where myself and the other Splunk admin were unable to access
the program’s logs. I have now resolved this problem, but recommend that the configuration file’s
access permissions be changed to prevent further unauthorized access as soon as possible.

Introduction: I recently encountered an issue with our Splunk security software affecting both me and
my fellow analyst Alice <surname>. Splunk, being the security information and event management
program we use at StackFull, records various logs from the firewall, Jira, SWE teams and other
sources. This is especially useful for us SOC analysts so if security incidents occur, we can examine all
logs needed in resolving it. However, when Alice gave me access to Splunk, we found that I was unable
to search any of its logs.

Details: On looking into this, Alice told me that one of Splunk’s configuration files named config.conf
had been edited by another employee, which was preventing both me and Alice from accessing Splunk
with admin permissions. In order to resolve this, I searched the /opt/splunk directory holding all
Splunk’s files, first trying the “tree” command which showed the contents without me needing to do a
search with “find”, and found the key config.conf file within it. I noticed (using “ls -l” on the directory
containing the file) a contributing factor, that the configuration file was currently editable by all users.
This is a security flaw which I will get back to, but for the moment I resolved the issue with log access
by editing the file to the state it should have (using the vim editor, opening it with “vi config.conf”),
adding Alice and myself back as Splunk administrators, and making a backup of the config file to my
own machine (with the command “cp config.conf /home/fstack”).

My recommendation would be that the Splunk configuration file’s permissions be changed as soon as
possible (using the “chmod” command) to prevent unauthorized edits, which I would be glad to take
care of if given approval for the task. Specifically, there are three categories of user that can be given
different permissions to read, write, or (not applicable here) execute a file: the file’s owner, the specific
group of users assigned to the file, and all other users. When I checked, the file had open permissions
for any and all users to edit or even delete it; an even bigger potential issue than the one I resolved, if
someone did delete it by mistake.

I would advise that specifically, the config file’s permissions are changed so that general users can read
it and do nothing else to it, keeping write permissions (which allow editing or deletion) provided only
to the owners and a user group containing level 2 analysts like myself and Alice. (E.g. “chmod 664
config.conf”) One tool which can be used to monitor the config file to ensure it stays in the state we
want is the md5sum Linux command, which computes an individualized code, also known as an MD5
hash, that serves to fingerprint a file. If the file is edited, the MD5 hash will change, as it had when I
checked before and after restoring admin permissions. My recommendation is that until further users
and administrators are given access to Splunk, we set up an automated program to check the config
file’s MD5 hash against the current hash, every day at the same time. This can be easily done with the
crontab command (e.g. in the crontab menu, “0 4 * * * if [md5sum /opt/splunk/ | != <current md5
hash>]; then echo “The splunk config file has been modified. Please check it.”) , and I would be happy
to get this set up as well, pending your authorization.

Conclusion: To summarize the above, the Splunk configuration file currently has a security flaw where
unauthorized users can change or delete it whenever they like. This already led to one issue, now
resolved, preventing myself and Alice <surname> from using our admin access. My recommendations
are that the file’s permissions be changed to make general users able to read but not write the config
file, and that the file’s current state be checked automatically each day with the md5sum command,
automated with a crontab process, both of which I could set up quickly and easily if authorized.

Thank you for your time, and I look forward to ensuring these issues will be prevented from arising
with Splunk in the future. - Alex Clark