Lab Guide 5

Wazuh-Elastic Training
Lab-Guide - Session 5
Wazuh 4.1.5
Elastic Stack 7.10.0
OpenDistro 1.12.0
Table of Contents
CDB Lists
Lab Exercise 5a - Escalate sshd alerts about known attacker IPs
Lab Exercise 5b - Distinguish between different ssh-watch-list values
Active Response
Active Response
Creating custom Active Commands:
Creating custom Active Response:
Stateful vs stateless command:
Lab Exercise 5c - deploy and test Linux firewall active response
Hacking Wazuh
Tools for Troubleshooting and Debugging Wazuh
Monitoring network communications between the manager and agents using
tcpdump
Monitoring files that are in-use by ossec processes using lsof (lsof)
Example:
Monitoring process communications between the agent and manager using strace
(for ossec-remoted)
Monitoring open sockets of Wazuh processes using netstat (or ss)
Docker Lab
Copyright © 2020 Wazuh, Inc. All rights reserved. 1

CDB Lists

CDB stands for “constant data base”. This acts as a high performance on-disk
associative array, mapping keys to values. They allow only two operations: creation and
reading. After the initial creation (compilation) of the CDB list, you can’t add any more data to
that file.
Use cases:
● Users (black or white lists)

● IP address lookups
● Indexing a list of bad domains
It’s also a convenient way to index databases and perform lookups on them.
CDB lists need to be stored in the directory /var/ossec/etc/lists/ and they need to
have a specific format. For example, file /var/ossec/etc/lists/suspicious could
contain entries like this:
key1:value
key2:value
key3:value
Each key has to be unique and must be followed by a colon and an optional value. The
values between lines can be the same as long as the key is unique.
We need to define it in the ossec.conf file first by adding the following entry to the <rule>
section of the ossec.conf file.
<ruleset>
<list>etc/lists/suspicious</list>
</ruleset>
The last thing to do is to create new rules to trigger alerts whenever an event matches one
of the entries in our new list.
<rule id="100001" level="10">

<if_sid>***SOME RULE***</if_sid>
<list field="srcip" lookup="address_match_key">lists/suspicious</list>
<description>Suspicious IP found</description>
</rules>

This rule will always trigger if the source IP is found in our CDB "suspicious" list.
Full documentation for all types of Wazuh CDB list lookups:

https://documentation.wazuh.com/current/user-manual/ruleset/cdb-list.html
Lab Exercise 5a - Escalate sshd alerts about known attacker IPs

For sshd invalid user login attempts where the attacker's IP is in our custom ssh-watch-list
file, cause a higher priority rule to fire than the standard rule 5710.
Example log entries

Jan 19 04:17:54 agent sshd[2365]: Failed password for invalid user blabla from 1.2.3.4 port
55969 ssh2
Jan 19 04:17:54 agent sshd[2365]: Failed password for invalid user root from 99.99.99.99
port 55969 ssh2
Jan 19 04:17:54 agent sshd[2365]: Failed password for invalid user admin from 10.10.10.10
port 55969 ssh2
Jan 19 04:17:54 agent sshd[2365]: Failed password for invalid user bill from 12.12.14.14
port 55969 ssh2
Initial Wazuh alert pattern for any of the above:
** Alert 1484799474.4188056: -
syslog,sshd,invalid_login,authentication_failed,pci_dss_10.2.4,pci_dss_10.2
.5,pci_dss_10.6.1,
2017 Jan 19 04:17:54 (linux-agent) 10.0.0.20->/var/log/secure
Rule: 5710 (level 5) -> 'Attempt to login using a non-existent user'
Src IP: 45.55.186.244
Jan 19 04:17:54 agent sshd[2365]: Failed password for invalid user blabla
from 55.55.186.244 port 55969 ssh2
We want to make a special rule of higher severity fire if this rule 5710 is tripped by an >
attacker on our special list of IPs of concern (ssh-watch-list)

Create /var/ossec/etc/lists/ssh-watch-list file with this content:
1.2.3.4:
55.55.186.244:
2.2.2.2:
5.5.5.5:
10.10.10.10:
23.32.23.32:
Add to the <ruleset> section of /var/ossec/etc/ossec.conf on the manager:

<list>etc/lists/ssh-watch-list</list>
Add this rule to /var/ossec/etc/rules/local_rules.xml
<rule id="100112" level="10">

<if_sid>5710</if_sid>
<list field="srcip" lookup="address_match_key">etc/lists/ssh-watch-list</list>
<description>SSH attacker on ssh-watch-list attempted to login using a non-existent
user</description>
</rule>
Feed the example log entries into /var/ossec/bin/ossec-logtest and observe that those
involving a listed IP fire rule 100112 instead of rule 5710. The higher severity of this rule could
be used to trigger a more severe active response action.
Lab Exercise 5b - Distinguish between different ssh-watch-list values

Let's maintain an ssh-watch-list that classifies attacker IPs as "bad" or "terrible". Make sshd
invalid user login attempts where the attacker's IP is listed in our custom ssh-watch-list file
with a value of "terrible", cause an even higher priority rule to fire than the already escalated
rule 100112.
Replace /var/ossec/etc/lists/ssh-watch-list file with this content and recompile:

1.2.3.4:bad
55.55.186.244:bad
2.2.2.2:terrible
5.5.5.5:bad
10.10.10.10:terrible
23.32.23.32:bad

Add this new rule to local_rules.xml, just previous to rule 100112.
<rule id="100111" level="15">

<if_sid>5710</if_sid>
<list field="srcip" lookup="match_key_value" check_value="terrible">etc/lists/ssh-watch-list</list>
<description>A truly terrible SSH attacker on ssh-watch-list attempted to login using a non-existent
user</description>
</rule>
Feed the example log entries into ossec-logtest and observe that those involving a listed IP
fire rule 100112 instead of rule 5710, unless they have a value of "terrible", in which case they
fire the even more severe rule 100111.
Lastly, you might like to see how Wazuh is incorporating some lists of its own directly into
the Wazuh Ruleset to make Amazon and audit rules easier to work with. This will get you
started exploring:
find /var/ossec/etc/lists
grep -r "etc/lists" /var/ossec/ruleset/rules
Also, to see a great blog on importing the OSINT list of malicious IPs into a CDB:
https://wazuh.com/blog/cdb-lists

Active Response

Active Response
In order to reduce risk and contain damage, we need in our system an automated
remediation to security violations and threats. A remediation solution can perform actions or
alerts configurations to prevent the access to the service or network where the threat came
from.
This automated remediation is called “Active response” in Wazuh. Active response allows
you to execute a script whenever a rule is matched by the ruleset. Any number of responses
can be attached to any of the rules, but it is important to be careful.
A bad implementation of the rules and responses, can be dangerous. An attacker could use
the rules against you.
By default the following active response scripts are already implemented with a default
Wazuh installation:
● disable-account.sh: disables an account by setting “passwd -l” (Linux, Solaris, AIX

only)
● firewall-drop.sh: adds an IP to the iptables deny list(Linux, Solaris, FreeBSD, AIX only)
● firewalld-drop.sh: adds an IP to firewalld drop list(Linux)
● host-deny.sh: adds an IP to the /etc/hosts.deny file (Linux, Solaris, FreeBSD, AIX only)
● ip-customblock.sh: Custom Wazuh block, easily modifiable for custom response
● ipfw_mac.sh: firewall-drop response script created for the Mac OS firewall
● ipfw.sh: firewall-drop response script created for ipfw, the FreeBSD IP packet filter
● pf.sh: firewall-drop response script created for pf, the BSD licensed stateful packet
filter.
● ossec-slack.sh: in order to post modifications
● restart-ossec.sh: automatically restarts Wazuh when ossec.conf has been changed
● route-null.sh: adds an IP to null route (only works on Linux and FreeBSD)
● route-null.cmd: adds an IP to null route (Windows Vista, Windows server 2008)
● route-null-2012.cmd: adds an IP to null route (Windows Server 2012)
● netsh.cmd: uses Windows "netsh ipsec" to block an IP - for Windows Server 2012
● netsh-win-2016.cmd: uses Windows "netsh advfirewall" to block an IP - for Windows
Server 2012 and 2016.

When creating custom Active Response scripts one should be careful and should always
test them before deploying them to production environments. These scripts are run as the
root user, and thus perform any action an administrator can do.
Creating custom Active Commands:
The scripts need a way to be referenced by the ruleset to perform a particular active
response. A command needs to be defined for this. Any script that can receive parameters
from the command line can be used for Active Response.
Active response commands are defined in the ossec.conf file.
Command Values:
Tags Values Description
Any name to define Used to reference the command from

name
the command other sections of the configuration
Name of the script to Name of the script or binary that will

execute be in charge of the response. It must
executable
exist in
/var/ossec/active-response/bin
Variables to be Comma-separated list of variables to

handed to the script be handed to the command line.
or executable Variables are in the order that is
expect specified here.
<expect /> means that the script

doesn’t expect any input
Yes / no If yes, the response will timeout

allowing an automated expiry of the
timeout_allowed
response. The executable must be
written to handle undoing its actions

Example:
<command>
<name>mail-test</name>
<executable>mail-test.sh</executable>
<timeout_allowed>no</timeout_allowed>
<expect />
</command>
In this example, the command is called “mail-test” and should execute the
mail-test.sh script located at /var/ossec/active-response/bin. It is not able to
timeout and the script doesn’t expect any input.
Let’s see another example:
<command>
<name>host-deny</name>
<executable>host-deny.sh</executable>
<expect>srcip</expect>
<timeout_allowed>yes</timeout_allowed>
</command>
In this case the command is called “host-deny”, and should execute the host-deny.sh,
the script needs the srcip and the command is able to timeout.
Creating custom Active Response:
Now we need to bind the command to one or more rules or a specific severity level, which
creates an active response. In the active response definition, it is declared when a
command is going to be executed and where, meaning on which environment (Agent,

Manager, Local, or everywhere).

The following tags can be used for active responses:
Tags Values Description
yes /no If yes, Active response is disabled.

disable
If no, Active response is enabled.
Must reference a Name of the command to be used in

command previously defined the response.
command
● local Define where the command is going to

● server be executed.
● defined-agent ● local: execute the command on
● all the agent that generated the
location event
● Server: execute the command
on the manager
Numeric identification Only used if the location is

agent_id
of your Wazuh agent defined-agent
Numeric value of the Command is executed for any rule

level severity matched where the level is this
severity or higher
Numeric rule ID Command is executed for any event

rules_id number matching one of the IDs. Multiple IDs
comma-separated.
Alphanumeric group Executed for any event matching a rule

rules_group names in one of these groups. Multiple groups
should be comma-separated.
Numeric value The duration in seconds , until the

timeout
action is reversed.

Example:
<active-response>
<command>mail-test</command>
<location>server</location>
<rules_id>1002</rules_id>
</active-response>
In this case, the mail-test command will be executed on the ossec manager everytime rule
1002 is fired. The rule 1002 description is: “Unknown problem somewhere in the system” and
is matched when a “bad word” is in the logs.
Bad words, is an array that is defined in the syslog_rules.xml and can be one of the
followings: core_dumped, failure, error, attack, bad, illegal, denied, refused,
unauthorized, fatal, failed, Segmentation Fault or Corrupted
They usually indicate that something is seriously wrong on a system.
Another example:
<active-response>
<command>host-deny</command>
<location>local</location>
<level>7</level>
<timeout>600</timeout>
</active-response>
In this example the host-deny command will be executed on the agent that reported the
rule-matching event, with rule levels equals or bigger than 7. This response has 600 seconds
for the timeout.

Stateful vs stateless command:
It’s important to understand the difference between a command being stateful and a
command being stateless. In our examples we saw one of each.
A Stateful command is a command that has the capability to remove the applied action
after a certain period of time.
To configure a command like this, we need to set the timeout_allowed option in the
command to yes, and then configure a timeout value in the active response to indicate how
long to wait until the applied action is automatically reversed.
An example for a Stateless command is the mail-test command we created.
A Stateless command is that command that doesn’t have the capability of removing the
applied changes that the response applied.
An example for a Stateful command is the host-deny command from above.

Lab Exercise 5c - deploy and test Linux firewall active response
Configure Wazuh to make you Linux agents firewall-block ssh brute force offenders..
1. On the manager, edit /var/ossec/etc/ossec.conf, inserting directly above the following

with the following -- Make sure to get rid of the comments lines, too: <!-- and —>
<active-response>
<command>firewall-drop</command>
<location>local</location>
<rules_id>5712</rules_id>
<timeout>120</timeout>
</active-response>
The command "firewall-drop" referenced above is already defined in ossec.conf on the

manager here as seen here below. You do not need to add it in.
<command>
<name>firewall-drop</name>
<executable>firewall-drop.sh</executable>
<expect>srcip</expect>
<timeout_allowed>yes</timeout_allowed>
</command>

When invoked by an active response, this command will call the
"/var/ossec/active-response/bin/firewall-drop.sh" script on the agent, passing an IP
number to it to be blocked via the local iptables firewall. Then after a timeout period it will
be called again to un-firewall the offending IP. If you wish, take a moment to locate and
examine this script. It can serve as an excellent template for you to build your own custom
active responses later.
2. Restart the Wazuh manager to apply this change:
systemctl restart wazuh-manager
3. From your linux-agent, perform an ssh dictionary attack against your elastic agent
LABSET=#
sshpass -p wrongPassWord ssh badguy@elastic$LABSET.lab.wazuh.info &
sleep .5
sleep .5
sleep .5
sleep .5
sleep .5
sleep .5
sleep .5
sleep .5
sleep .5
sleep .5
sleep .5
sleep .5
sleep .5
sleep .5
sleep .5

4. Notice that while initially there are multiple repetitions of "Permission denied, please try
again" in the output, that the last several sshpass attempts get no reply, because your elastic
system firewall started blocked your linux agent via its firewall. If you now run "ssh
wazuh@elastic1.lab.wazuh.info", it will just stall. Hit ctrl-c to cancel that attempt.
5. Clean up the stalled backgrounded sshpass sessions with "killall sshpass"
6. After the 120 second timeout passes, we should also be able to see the record of the
active response being performed and then later reversed on your linux-agent:
Inspect the AR log via Kibana with this search: decoder.name:"ar_log"
7. Confirm that your linux-agent is again able to reach your elastic system.
# ssh wazuh@elastic#.lab.wazuh.info
wazuh@elastic#.lab.wazuh.info's password:
...
8. Check Kibana later today for active response activity (see step 6) and you will probably
see real attackers in the wild being blocked by one or more of your Linux lab systems.

Hacking Wazuh

Tools for Troubleshooting and Debugging Wazuh
There are 4 power-tools on UNIX-based systems when it comes to troubleshooting or

debugging a problem. Be it either a faulty application that keeps throwing segfaults or
performance issues.
Those 4 tools are:
● tcpdump
● lsof
● strace
● netstat
tcpdump(1) is the swiss-army knife when it comes to troubleshooting all sorts of network
issues, e.g if for some reason an agent suddenly loses the connection to the manager and
wouldn’t connect. In a complex environment, with lots of firewalls or NIPS in between,
tcpdump is a very powerful tool. Below is a detailed example on how we can analyze
network traffic to localize a potential bottleneck in the network communication.
lsof(8) is another diagnostic tool that lists all files of processes running on a specific host that
are currently in use.
strace(1) is useful for monitoring system calls between the manager and agents. Since only
the network traffic between the devices is encrypted through the Wazuh message protocol,
we are able to use strace to read those syscalls and exchange messages in plaintext. Below
there is a detailed described example on how we can use strace to debug a problem on the
manager or on the agents.
netstat(8) lets you print open network connections and routing tables. In our case we will
use netstat to display the open sockets to the corresponding processes.

Monitoring network communications between the manager and agents
using tcpdump
We are going to use the following options for debugging our network issue. With the flag
-nn we specify that we don’t want to resolve hostnames or ports, instead we want to use the
IP address.
The -i flag specifies the interface we want tcpdump to capture data from, in our case we
specify that only traffic from our interface eth0 should be captured. We also want to
configure tcpdump to only listen on the Wazuh port, 1514, by using the -p flag.
By using the -AA (must be all caps) we tell tcpdump to display the content in ASCII (often
used for capturing web pages).
Finally, we also want to specify the size of the whole packet, in our case we select 0 , we do
this by using the -s flag.
[root@manager ~]# tcpdump -nni eth0 port 1514 -AA -s 0

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
16:23:57.446537 IP 10.0.0.4.59075 > 10.0.0.35.1514: UDP, length 145
.X.T/..E.." ..E.....@.@..-
...
..JFD&.#5..r..C..j.h......Bs...N.y.I.{'..I.+.7;....... \Z~.....r.n..2
.<..8a..l.=q.:.}~2C.
...
..#.....9..:....H.R..f..dW.twF.R....2..2.q.G.^./.M....4..:.&
.....~.............4.,lA....3...[........D.s...e...dq.*.&f......?g:..C....2.z8.H-..
.$h.G..9A.qb..
%B._..$I\
..~.,..\.....6..c.48... kb.....3.......s.3T.....e$..&.|3..P..#;s.h..wf
..@.F..0:....Z0
16:29:14.015574 IP 10.0.0.91.36478 > 10.0.0.35.1514: UDP, length 294

.X.T/..L..z...E..B..@.@...
..[
..#.~......!001!:!|. .].iM(j.l.T..>==54wVE[=`.c.......C......
Z..,.-.................T.K.
D.f`Z..d....{m..Sc\M....L......].w..JM.:.,.%.......?..)..6.D.
Z....XWc....~...........w$z.n>|:...&8...Z$.J.......7x.A;...
....!.b.n...QE.....d.<E#V!...?W
we..../.f/*.....$.x..>$....hQ.#.....z.p.9....fq;).2O....l..~.
We can see that pretty much the entire network communication is some gibberish output
due to the encryption, almost…. We can still extract some useful information. In order to

understand what Wazuh is doing there, we need to take a quick look at the client.keys
file:
[root@manager ~]# cat /var/ossec/etc/client.keys

001 indexer any 4d2a6189ac83b470793cfa3f4621057acd8720d077085a3e5bc5eaa08fadc767
003 windowsAgent 10.0.0.126
4c84dbacaba317a724ec651482fc3ea07ccc7d210e869386fa5c05e190dde53c
005 linuxAgent 10.0.0.4
673bc4d694a34e50571ed8c00e9c6edd301ab327cb0ebd18bce885f5b84ff89a
Looking at the client.keys file, we can see that one agent (indexer) is on a dynamic ip
address (dhcp) therefore it is connected to the manager through “any” and for the Linux
Agent we have a dedicated static IP address, 10.0.0.4
This is essential, because the ossec-remoted process is looking for a key to decrypt the
message. The goal is to find the right corresponding key, therefore the remoted process
opens the client.key files and looks either for an IP address (if specified) or when only “any” is
used, it looks for the agent ID, which displayed between two exclamation marks !001!
The rest of the captured message is useless to us, because its content is encrypted. This
was just to demonstrate what the encrypted network communication between the manager
and the agents looks like.

Monitoring files that are in-use by ossec processes using lsof (lsof)
lsof(8) can be performed on any regular file, directory, library, stream or network file (e.g
internet socket, nfs file, or UNIX domain socket).
In this particular case, we want to take a closer look at various log files which are configured
in our ossec.conf (for log analysis) that are in use by different processes.
Example:
[root@manager ~]# lsof /var/log/messages

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
rsyslogd 501 root 3w REG 202,1 16867 17126018 /var/log/messages
ossec-log 20146 root 5r REG 202,1 16867 17126018 /var/log/messages
This output shows that the rsyslog daemon, responsible for logging the main system/kernel
events on Linux systems, writes its log messages into the messages file. In the FD (file
descriptor) column this is shown as “w” for write access. Whereas the ossec-logcollector
process is only reading its log messages and sending the content through a unix domain
socket to the analysisd process for normalization and decoding.

Monitoring process communications between the agent and manager
using strace (for ossec-remoted)
strace(1) can be used to trace system calls and signals when trying to diagnose and
troubleshoot problems in the communication between the manager and the agent. In this
particular case we want to take a closer look at the ossec-remoted process.
Each line in the trace contains the system call name, followed by its arguments in
parentheses and its return value. An example from stracing the command ``cat /dev/null”” is:
open("/dev/null", O_RDONLY) = 3
Errors (typically a return value of -1) have the errno symbol and error string appended.
open("/foo/bar", O_RDONLY) = -1 ENOENT (No such file or directory)
Signals are printed as a signal symbol and a signal string. An excerpt from stracing and
interrupting the command ``sleep 666'' is:
sigsuspend([] <unfinished ...>
--- SIGINT (Interrupt) ---
+++ killed by SIGINT +++
Now we want to trace the process ossec-remoted on the manager, by issuing the following
command:
[root@manager# ~]# strace -ff -o log -s 20000 -p $(pidof ossec-remoted)
The flag -o filename will write the output of the trace into a file called log, rather than to
stderr
The flag -ff specifies that each process is written to filename.pid where pid is the numeric
process id of each process.
The flag -s specifies the maximum string size to print, the default value is 32.
The flag -p specifies the process id of the process that we want to trace.

Since we write the output of the trace into a file, we need to take a closer look at the file
“log”.
strace(1) is creating a separate file for each fork process that belongs to ossec-remoted,
therefore we have 3 separate files
log.30574
log.30579
log.30580
Let’s take a closer look at the main process, log.30574
[root@manager# ~]# cat log.30574

recvfrom(4,
":P\237M_\375\307\tt~'\354\20*\324\376\222X\3\207\377\327\355Qq\222O\306\n6\346\0\3
63\250\302X\311\301\332\21|\231|\215\203\221X\22\224\310\33\274
\337\225\20\232\224\214xJ\370\373\245\323D\277\203\276^\234\371n\233\6\221\22\2448o
\334\340\350a\362\5\4=\353\31#\10\342E\243D\216\236\223
\221\225\35\247\24\244\234\244\341\0b\24\0\246{.\215\334\224$\3=;\261\207\355\252\3
\212q\331D\275\236\236\371\0\243H\244\262\213$\340\t\341d\205\324\20\233\206\255v\2
11\307\356\370kKo\34\266h\27\27\366\376\260\r~\225\265w\267\204\363i\335\327u6\371\
16\v4\243g\241\335\205\334\342\334\331\276|\354&\224\374\250/\314\256\243\243\344\2
21\31\327\265\2@@()\344\22\202\211\371K\224\34\370\225^f\213
@\nc\255\2J\252j\231\311", 6144, 0, {sa_family=AF_INET, sin_port=htons(38025),
sin_addr=inet_addr("10.0.0.4")}, [16]) = 241
brk(0) = 0x1bc5000
brk(0x1bf5000) = 0x1bf5000
brk(0) = 0x1bf5000
brk(0) = 0x1bf5000
brk(0x1be5000) = 0x1be5000
brk(0) = 0x1be5000
brk(0) = 0x1be5000
brk(0) = 0x1be5000
brk(0x1bd5000) = 0x1bd5000
brk(0) = 0x1bd5000
brk(0) = 0x1bd5000
brk(0) = 0x1bd5000
brk(0x1bc5000) = 0x1bc5000
brk(0) = 0x1bc5000
write(10, "1:7313:", 7) = 7
lseek(10, 0, SEEK_SET) = 0
sendto(4,
":=\317Y\254E\261\310\347\v\37\3504\236\7\35\266F\255(\316\233\f\252\235\373\234o|\
217\33\17f$\211\220\312\264\345\273\235I\v$g\270\351\202\231\315:\257\215b\353j\272

f\347\275\215\277\21M]\n\371U\363\247\334\214+", 73, 0, {sa_family=AF_INET,
sin_port=htons(38025), sin_addr=inet_addr("10.0.0.4")}, 16) = 73
open("/queue/agent-info/linuxAgent-10.0.0.4", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 12
fstat(12, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0x7f59450d7000
write(12, "Linux linuxagent 3.10.0-327.10.1.el7.x86_64 #1 SMP Sat Jan 23 04:54:55
EST 2016 x86_64 - OSSEC HIDS v2.8 / f821fbaeeed8a954980540297cec7707\n", 140) = 140
close(12) = 0
munmap(0x7f59450d7000, 4096) = 0
futex(0x634484, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x634480, {FUTEX_OP_SET, 0,
FUTEX_OP_CMP_GT, 1}) = 1
futex(0x6344c0, FUTEX_WAKE_PRIVATE, 1) = 0
recvfrom(4, <detached ...>
In this trace we see that ossec-remoted is fetching the following information from the
connected agent
● ip address (in this case 10.0.0.4)

● the output of uname -a, including hostname, kernel version, date,
architecture, Wazuh version and the md5sum of the agent.conf

Monitoring open sockets of Wazuh processes using netstat (or ss)
netstat(8) or respectively the more modern ss(8) can be used to investigate sockets or
open network connections. We are going to focus on the open sockets, especially the UNIX
domain socket that is used as a message queue to exchange data between the respective
Wazuh processes.
The following commands can be used to display this, by setting the right flags.
[root@manager# ~]# netstat -nap |grep ossec
respectively
[root@manager# ~]# ss -nap |grep ossec
For those more comfortable with netstat, can of course use this, but it shall be noted that it’s
considered deprecated, meaning that it may be dropped by various Linux distributions at
some point. Let’s take a closer look at those flags.
-n shows numerical addresses instead of resolving service names

-a displays both listening and non-listening sockets (established
connections)
-p shows the PID and name of a program to which each socket belongs.
[root@manager ~]# netstat -nap |grep ossec
udp 0 0 0.0.0.0:1514 0.0.0.0:*
20153/ossec-remoted
unix 7 [ ] DGRAM 1791185 20142/ossec-analysi
/queue/ossec/queue
unix 3 [ ] DGRAM 1791208 20153/ossec-remoted
/queue/alerts/ar
unix 3 [ ] DGRAM 1791174 20139/ossec-execd
/var/ossec/queue/alerts/execq
unix 2 [ ] DGRAM 1791204 20153/ossec-remoted

unix 2 [ ] DGRAM 1791214 20146/ossec-logcoll
unix 2 [ ] DGRAM 1791231 20162/ossec-monitor
unix 2 [ ] DGRAM 1791213 20156/ossec-syschec
unix 2 [ ] DGRAM 1791192 20156/ossec-syschec
[root@manager ~]# ss -nap |grep ossec
u_dgr UNCONN 0 0 /queue/ossec/queue 1791185 * 0
users:(("ossec-analysisd",pid=20142,fd=3))
u_dgr UNCONN 0 0 /queue/alerts/ar 1791208 * 0
users:(("ossec-remoted",pid=20153,fd=11))
u_dgr UNCONN 0 0 /var/ossec/queue/alerts/execq 1791174
* 0 users:(("ossec-execd",pid=20139,fd=3))
u_dgr UNCONN 0 0 * 1791212 * 1791174
u_dgr UNCONN 0 0 * 1791204 * 1791185
u_dgr UNCONN 0 0 * 1791214 * 1791185
users:(("ossec-logcollec",pid=20146,fd=4))
u_dgr UNCONN 0 0 * 1791211 * 1791208
u_dgr UNCONN 0 0 * 1791231 * 1791185
users:(("ossec-monitord",pid=20162,fd=3))
u_dgr UNCONN 0 0 * 1791213 * 1791185
users:(("ossec-syscheckd",pid=20156,fd=4))
u_dgr UNCONN 0 0 * 1791192 * 1791185
users:(("ossec-syscheckd",pid=20156,fd=3))
udp UNCONN 0 0 *:1514 *:*

We particularly want to focus on the ossec-remoted process, as it is this process that
interacts with the individual processes on the manager and the agents. The remoted
process is receiving data through a message queue (socket) from the agentd process on the
client-side but it is also sending data by triggering remote actions like a syscheck and
rootcheck run triggered from the manager.
The socket / message queue is located in /var/ossec/queue/ossec/queue
The below graphic should illustrate this better:

Docker Lab
On elastic instance
yum -y install docker

# install Python docker API module which in CentOS/RedHat7+ is better acquired via "yum install python-docker-py"
yum -y install python-pip
pip install docker
systemctl start docker
Add to mgr's linux group agent.conf
<agent_config name="elastic#">
<wodle name="docker-listener">
<disabled>no</disabled>
</wodle>
</agent_config>
On elastic
docker pull nginx

docker run -d --name ngtest nginx
docker pause ngtest
docker unpause ngtest
docker exec -it ngtest /bin/bash
(then exit out)
docker stop ngtest
docker rm ngtest
Search Kibana to see history of Docker activity
data.integration:docker

Lab Guide 5

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Lab Guide 5

Uploaded by

Copyright:

Available Formats

Wazuh-Elastic Training

Copyright © 2020 Wazuh, Inc. All rights reserved. 1

Copyright © 2020 Wazuh, Inc. All rights reserved. 2

● Users (black or white lists)

<rule id="100001" level="10">

Copyright © 2020 Wazuh, Inc. All rights reserved. 3

Full documentation for all types of Wazuh CDB list lookups:

Lab Exercise 5a - Escalate sshd alerts about known attacker IPs

Example log entries

Initial Wazuh alert pattern for any of the above:

Copyright © 2020 Wazuh, Inc. All rights reserved. 4

Add to the <ruleset> section of /var/ossec/etc/ossec.conf on the manager:

Add this rule to /var/ossec/etc/rules/local_rules.xml

<rule id="100112" level="10">

Lab Exercise 5b - Distinguish between different ssh-watch-list values

Replace /var/ossec/etc/lists/ssh-watch-list file with this content and recompile:

Copyright © 2020 Wazuh, Inc. All rights reserved. 5

<rule id="100111" level="15">

Copyright © 2020 Wazuh, Inc. All rights reserved. 6

Copyright © 2020 Wazuh, Inc. All rights reserved. 7

● disable-account.sh: disables an account by setting “passwd -l” (Linux, Solaris, AIX

Copyright © 2020 Wazuh, Inc. All rights reserved. 8

Creating custom Active Commands:

Active response commands are defined in the ossec.conf file.

Tags Values Description

Any name to define Used to reference the command from

Name of the script to Name of the script or binary that will

Variables to be Comma-separated list of variables to

<expect /> means that the script

Yes / no If yes, the response will timeout

Copyright © 2020 Wazuh, Inc. All rights reserved. 9

mail-test.sh script located at /var/ossec/active-response/bin. It is not able to

timeout and the script doesn’t expect any input.

Let’s see another example:

Creating custom Active Response:

creates an active response. In the active response definition, it is declared when a

command is going to be executed and where, meaning on which environment (Agent,

Copyright © 2020 Wazuh, Inc. All rights reserved. 10

Tags Values Description

yes /no If yes, Active response is disabled.

Must reference a Name of the command to be used in

● local Define where the command is going to

Numeric identification Only used if the location is

Numeric value of the Command is executed for any rule

Numeric rule ID Command is executed for any event

Alphanumeric group Executed for any event matching a rule

Numeric value The duration in seconds , until the

Copyright © 2020 Wazuh, Inc. All rights reserved. 11

Copyright © 2020 Wazuh, Inc. All rights reserved. 12

An example for a Stateless command is the mail-test command we created.

An example for a Stateful command is the host-deny command from above.

Copyright © 2020 Wazuh, Inc. All rights reserved. 13

1. On the manager, edit /var/ossec/etc/ossec.conf, inserting directly above the following

The command "firewall-drop" referenced above is already defined in ossec.conf on the

Copyright © 2020 Wazuh, Inc. All rights reserved. 14

2. Restart the Wazuh manager to apply this change:

systemctl restart wazuh-manager

Copyright © 2020 Wazuh, Inc. All rights reserved. 15

5. Clean up the stalled backgrounded sshpass sessions with "killall sshpass"

Copyright © 2020 Wazuh, Inc. All rights reserved. 16

Copyright © 2020 Wazuh, Inc. All rights reserved. 17

There are 4 power-tools on UNIX-based systems when it comes to troubleshooting or

udp UNCONN 0 0 :1514 :*