Splunk® User Behavior Analytics

Release Notes 5.2.0

Generated: 7/22/2023 11:50 pm

Copyright (c) 2023 Splunk Inc. All Rights Reserved

 Table of Contents
Splunk User Behavior Analytics Release Notes................................................................................................................1
Welcome to Splunk UBA 5.2.0..................................................................................................................................1
Known issues in Splunk UBA....................................................................................................................................2
Fixed issues in Splunk UBA.......................................................................................................................................9
Log4j in Splunk UBA 5.2.0.......................................................................................................................................10

Where to find help..............................................................................................................................................................11

Getting help with Splunk UBA..................................................................................................................................11

Third-party software...........................................................................................................................................................12
Third-party credits in Splunk UBA............................................................................................................................12

i
Splunk User Behavior Analytics Release Notes

Welcome to Splunk UBA 5.2.0

Splunk UBA 5.2.0 is a major release. See About Splunk User Behavior Analytics and release types for more information
about the different types of Splunk UBA releases.

If you are new to Splunk UBA, review all the steps in the Splunk UBA installation checklist before installing Splunk UBA.

Planning to upgrade from an earlier version?

If you plan to upgrade to this version from an earlier version of Splunk UBA, read the following documents before you get
started:

• See Upgrade Splunk UBA prerequisites and overview in the Install and Upgrade Splunk user Behavior Analytics
manual for information you need to know before you upgrade.
• Splunk UBA requires incremental upgrades from earlier versions. See How to install or upgrade to this release of
Splunk UBA in the Install and Upgrade Splunk User Behavior Analytics manual for upgrade path information.

What's new in 5.2.0

Splunk UBA version 5.2.0 includes the following features and changes:

Feature, enhancement, or
Description
change
The 5.2.0 release provides the following operating system updates:

• Support for Ubuntu version 20.04 (new installations and upgrades).

• Support for RHEL version 8.6 (new installations and upgrades).
• Support for Oracle/Linux version 8.7 (new installations and upgrades).
Operating System updates:
Caution: The 5.2.0 AMI package will be available shortly after GA for AWS environments.

For more information, see Operating system requirements in the Install and
Upgrade Splunk User Behavior Analytics manual.
Bulk upload users to User Watchlists You can now add users in bulk to a User Watchlist. See, Add bulk users to a User Watchlist.

You can now enter individual entries within the Allow/Deny by adding values using the Splunk UBA
Allow/ Deny List improvement
user interface. See, Add new entries to a deny list or allow list.

Delete multiple threats of a certain type The clean_threats.sh script can be used to clean up old threats. See, How to delete multiple
based on the score threats of a certain type based on the score.

The Hypergraph based Malware Threat Detection Model that was disabled in version 5.1.0 is
Malware Threat model re-enabled
re-enabled in version 5.2.0.

Removal of biased language As part of an ongoing process across releases, user-interface mentions of the terms "blacklist" and
"whitelist" are changed as follows:

• The term "blacklist" has changed to "deny list".

• The term "whitelist" has changed to "allow list".

1
 Feature, enhancement, or
Description
change
For more information, see Biased Language Has No Place in Tech
Splunk UBA external dependencies

You can download a PDF file listing the external dependencies required to install Splunk UBA:

• Splunk UBA External Dependencies

Do not independently upgrade the following UBA-dependent components to avoid impacting UBA operations:

• docker
• hadoop
• hive
• impala
• influxdb
• kubernetes
• nodejs
• openjdk
• postgresql
• redis
• spark
• zookeeper

Known issues in Splunk UBA

This version of Splunk UBA has the following known issues and workarounds.

If no issues are listed, none have been reported.

Date Issue
Description
filed number
Upon applying the Ubuntu security patches, postgresql got removed causing UBA unable to start

Workaround:
Stop all UBA Services :
/opt/caspida/bin/Caspida stop-all

Re-install postgres package, replace <uba ext packages> with your package folder in below command. Fo
uba-ext-pkgs-5.0.5 :
2023-06-08 UBA-17446

sudo dpkg --force-confold --force-all -i /home/caspida/<Extracted uba external package folder>

Start all UBA Services :

/opt/caspida/bin/Caspida start-all

2
 Date Issue
Description
filed number
Error after enabling Splunk SSL certificate validation - Connection refused , Splunk host validation error: %s

Workaround:
1. Run the below sed command into the UBA system.
sed --in-place=".bak" 's/validateSplunkHost(hostname/validateSplunkHost(hostname + ":" + this.
/opt/caspida/web/caspida-ui/server/security/splunkLoginProvider.js

2. Edit (if not already done) file "/etc/caspida/local/conf/uba-site.properties" add/modify property "validate.s
3. Run the sync-cluster command

/opt/caspida/bin/Caspida sync-cluster /etc/caspida/local/conf/

UBA-17437,
2023-06-06
UBA-17455
4. Restart the caspida-jobmanager

sudo service caspida-jobmanager stop

sudo service caspida-jobmanager start

5. Restart the caspida-ui

/opt/caspida/bin/Caspida stop-ui
/opt/caspida/bin/Caspida start-ui

Threat dashboard throwing error "Cannot read properties of undefined (reading 'length')" upon changing time to local time instea

Workaround:
1. Move to directory as mentioned in below command
cd /opt/caspida/web/zplex/server/databases/postgres

2. Replace code snippet in postgres.js file which will handle the issue using below command

sed --in-place=".bak" 's/result = result.rows;/if (Array.isArray(result)) {result = [].conc

rs.rows));} else {result =
result.rows;}/' /opt/caspida/web/zplex/server/databases/postgres/postgres.js
2023-05-24 UBA-17335

3. Stop Caspida UI

/opt/caspida/bin/Caspida stop-ui

4. Start Caspida UI

/opt/caspida/bin/Caspida stop-ui

2023-05-03 UBA-17233 Model execution failure caused by ModelRegistry.json not being applied correctly for deployments with 7 nodes or more after up

Workaround:
Note: Apply the following workaround to deployments with 7 nodes or more.

3
 Date Issue
Description
filed number
1) SSH as caspida to UBA management node (node 1)

2) Back up original file:

cp /opt/caspida/content/Splunk-Standard-Security/modelregistry/offlineworkflow/ModelRegistry.j
/opt/caspida/content/Splunk-Standard Security/modelregistry/offlineworkflow/ModelRegistry.jso

3) Copy the ModelRegistry.json.large_deployment file to the ModelRegistry.json file:

cp /opt/caspida/content/Splunk-Standard-Security/modelregistry/offlineworkflow/ModelRegistry.j
/opt/caspida/content/Splunk-Standard-Security/modelregistry/offlineworkflow/ModelRegistry.jso

4) Sync cluster:

/opt/caspida/bin/Caspida sync-cluster /opt/caspida/content/Splunk-Standard-Security/modelregis

5) Restart all services:

/opt/caspida/bin/Caspida stop-all
/opt/caspida/bin/Caspida start-all

Restricted sudo access for caspida ubasudoers file missing permissions

Workaround:
Run the following commands:
sed -i '120i\ /usr/sbin/service cri-docker *, \\' /opt/caspida/etc/sudoers.d/ubasudo
sed -i '130i\ /sbin/service cri-docker *, \\' /opt/caspida/etc/sudoers.d/ubasudoers
sed -i '135i\ /bin/systemctl start kubelet.service, /usr/bin/systemctl start kubelet
2023-04-20 UBA-17188 /opt/caspida/etc/sudoers.d/ubasudoers
sed -i '135i\ /bin/systemctl restart kubelet.service, /usr/bin/systemctl restart kub
/opt/caspida/etc/sudoers.d/ubasudoers
sed -i '135i\ /bin/systemctl start docker.service, /usr/bin/systemctl start docker.s
/opt/caspida/etc/sudoers.d/ubasudoers
sed -i '135i\ /bin/systemctl restart docker.service, /usr/bin/systemctl restart dock
/opt/caspida/etc/sudoers.d/ubasudoers
/opt/caspida/bin/Caspida sync-cluster /opt/caspida

2023-04-14 UBA-17151 UBA backup script fails if Redis network connection password changed from default

Workaround:
To complete a UBA backup, temporarily disable the redis password and then re-enable it afterwards

1. Stop Caspida

/opt/caspida/bin/Caspida stop

2. Open the file /etc/caspida/local/conf/custom/splunkuba-redis.conf and comment out the line requirepass

3. Sync the file across the cluster

/opt/caspida/bin/Caspida sync-cluster /etc/caspida/local

4. Start Caspida

4
 Date Issue
Description
filed number
/opt/caspida/bin/Caspida start

5. Perform uba backup

6. Go through steps 1-4 again, but for step 2, uncomment the requirepass setting
ImagePullBackOff Failure: Container workers sometimes gets removed from management node iptables after upgrade on AWS

Workaround:
The other nodes that run the containers should be re-added back to the iptables on the management node. Run the following on
sudo iptables -I DOCKER-USER -s <nodes> -i <external_interface> -j ACCEPT

Where <nodes> is the list of comma separated nodes that runs docker images. Get the list of worker nodes
2023-04-13 UBA-17148
a) grep -w container.worker.host /etc/caspida/conf/deployment/caspida-deployment.conf | cut -d"

b) Get the impala host by running: grep -w impala.server.host /etc/caspida/conf/deployment/caspida-

-d"=" -f2

And then <external_interface> is the output of route | grep default | awk '{print $8}'.
2023-04-05 UBA-17117 TLS/SSL Weak Message Authentication Code Cipher Suites

Workaround:

For 443 port

1. Stop the caspida-ui

sudo service caspida-ui stop

2. Open uiConfig.js file

vi /opt/caspida/web/caspida-ui/server/config/uiConfig.js

3. Find the cipher string

/ciphers

4. The string will look like this

ciphers: "AES128-GCM-SHA256:!RC4:HIGH:!MD5:!aNULL:!EDH:!DES:!3DES",

5. Add the following ciphers with the negate condition to deny the ciphers

ciphers: "AES128-GCM-SHA256:!RC4:HIGH:!MD5:!aNULL:!EDH:!DES:!3DES:!CBC:!SHA",

6. Save the file

7. Start the caspida-ui

sudo service caspida-ui start

5
Date Issue
Description
filed number

For 10250 port

1. Stop the containers

/opt/caspida/bin/Caspida stop-containers

2. Open kubeadm-conf.yaml.template file

vi /opt/caspida/conf/containerization/kubeadm-conf.yaml.template

3. Find the tlsCipherSuites string

/tlsCipherSuites

4. The string will look like this

tlsCipherSuites: [ TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA38

TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_G
TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA ]

5. Remove the desired ciphers

tlsCipherSuites: [ TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA38

TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_GCM_SHA384 ]

6. Find the tls-cipher-suites string

/tls-cipher-suites

7. The string will look like this

tls-cipher-suites:
"TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WIT
_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_GCM_SHA38
_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA"

8. Remove the desired ciphers suits

tls-cipher-suites:
"TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_
_RSA_WITH_AES_256_GCM_SHA384"

9. Save the file

10. Remove containerization

/opt/caspida/bin/Caspida remove-containerization

11. Setup containerization

/opt/caspida/bin/Caspida setup-containerization

6
 Date Issue
Description
filed number
12. Stop all services

/opt/caspida/bin/Caspida stop-all

13. Start all services

/opt/caspida/bin/Caspida start-all

MalwareThreatDetectionModel Failure: Spark-default.conf not updated under /var/vcap/packages/spark/conf directory after upgr
5.2.0

Workaround:
Note: For those upgrading UBA from 5.1.0.x to 5.2.0, apply the following workaround. This is not needed if upgrading from UBA

2023-04-03 UBA-17106
Update the spark configuration by running the following command on all the nodes:

cp -v /opt/caspida/conf/spark/spark-defaults.conf /var/vcap/packages/spark/conf/spark-defaults
/opt/caspida/bin/Caspida stop-spark
/opt/caspida/bin/Caspida start-spark

2023-02-02 UBA-16909 "UBA_HADOOP_SIZE key is either missing or has no value" error encountered when uba-restore is run from a backup created

2023-02-01 UBA-16900 Impala query to populate "Data Transfer by Source Device" panel from UBA Dashboards screen gets timeout.

Kubelet unable to fetch container log stats for inactive pods

2023-01-31 UBA-16886
Workaround:
Upgrade to UBA 5.3.0, which contains the fix from Kubernetes.

2023-01-25 UBA-16850 Getting error about zookeeper-server service is not responding in Health monitor UI intermittently (Ubuntu only)

UBA UI not accessible after performing RHEL8 post-upgrade clean up tasks

Workaround:
1) On all UBA nodes re-install missing redis package: sudo yum install redis-5.0.3-5*
2023-01-18 UBA-16818
2) Stop-all then Start-all UBA services:

/opt/caspida/bin/Caspida stop-all /opt/caspida/bin/Caspida start-all

UBA Spark Server failing to store failed model execution record in redis

Workaround:
On the spark master, run this command:
sed -i '7s|$|:/var/vcap/packages/spark/jars/*|' /opt/caspida/bin/SparkServer
2023-01-18 UBA-16819
Then, restart spark from the management node

/opt/caspida/bin/Caspida stop-spark
/opt/caspida/bin/Caspida start-spark

2023-01-05 UBA-16762 Benign JNDI ClassNotFoundException: Impala queries complain since removing Log4j2 vulnerability

2022-12-22 UBA-16722 Error in upgrade log, /bin/bash: which: line 1: syntax error: unexpected end of file

2022-12-05 UBA-16617

7
 Date Issue
Description
filed number
Repeated Kafka warning message "Received a PartitionLeaderEpoch assignment for an epoch < latestEpoch. This implies mes
order"

Workaround:
1) On zookeeper node (typically node 2 on a multi-node deployment), find all leader-epoch-checkpoint files: locate leader-e
also use a find command if locate isn't available)

a) Copy result into a script, adding ">" prior to each result. i.e.

#!/bin/bash
> /var/vcap/store/kafka/AnalyticsTopic-0/leader-epoch-checkpoint
> /var/vcap/store/kafka/AnalyticsTopic-1/leader-epoch-checkpoint
> /var/vcap/store/kafka/AnalyticsTopic-10/leader-epoch-checkpoint
> /var/vcap/store/kafka/AnalyticsTopic-11/leader-epoch-checkpoint
... b) Make script executable: chmod +x <script name>.sh 2) On node 1, run: /opt/caspida/bin/Caspida stop-al
run: ./<script name>.sh 4) On node 1, run: /opt/caspida/bin/Caspida start-all 5) Check logs to see if warn mes
zookeeper node: tail -f /var/vcap/sys/log/kafka/server.log

6) If you see the following warning repeated:

WARN Resetting first dirty offset of __consumer_offsets-17 to log start offset 3346 since the
3332 is invalid. (kafka.log.LogCleanerManager$) a) Clear cleaner-offset-checkpoint on zookeeper node by runnin
/var/vcap/store/kafka/cleaner-offset-checkpoint b) Then on node 1, run: /opt/caspida/bin/Caspida stop
/opt/caspida/bin/Caspida start-all

2022-07-26 UBA-15997 Benign error messages on clean UBA install: Relations do not exist, Kafka topic does not exist on ZK path

krb5-libs(x86-64) = 1.18.2-* is needed by krb5-devel-1.18.2-* on Oracle Enterprise Linux and RHEL

Workaround:
[5.1.0/5.1.0.1]

krb5-libs is required by the OS and cannot be removed. It must match the version of krb5-devel. If you hav
the latest krb5-devel.

1. sudo yum install krb5-devel

2. Rerun the INSTALL.sh command

2022-07-19 UBA-15963 If you do not have internet access and are okay with a lower version you can force a downgrade by runnin

1. sudo yum -y localinstall /home/caspida/Splunk-UBA-5.1-Packages-RHEL-8/extra_packages/rpm/hadoop/krb5-libs-1.18

2. Rerun the INSTALL.sh command

[5.2.0]

1. sudo yum -y localinstall /home/caspida/Splunk-UBA-5.2-Packages-RHEL-8/extra_packages/rpm/hadoop/krb5-libs-1.18

2. sudo yum -y localinstall /home/caspida/Splunk-UBA-5.2-Packages-RHEL-8/extra_packages/rpm/hadoop/zlib-1.2.11-20
3. Rerun the INSTALL.sh command

Upgrade from 5.0.5.1 to 5.1.0 or 5.2.0 (RHEL) OutputConnector re-import cert needed
2022-06-30 UBA-15912
Workaround:
import cacert again

8
 Date Issue
Description
filed number
Streaming model state will be reset on upgrade from 5.0.x to 5.1.0 or 5.2.0 - will require one week or more of data ingest to see
2022-06-23 UBA-15885
(Error message: Failed to deserialize object in InputStream)

2021-08-30 UBA-14755 Replication.err logging multiple errors - Cannot delete snapshot s_new from path /user: the snapshot does not exist.

Kubernetes certificates expire after one year

Workaround:
Run the following commands on the Splunk UBA master node:
2020-04-07 UBA-13804
/opt/caspida/bin/Caspida remove-containerization
/opt/caspida/bin/Caspida setup-containerization
/opt/caspida/bin/Caspida stop-all
/opt/caspida/bin/Caspida start-all

Backend anomaly and custom model names are displayed in Splunk UBA
2019-10-07 UBA-13227
Workaround:
Click the reload button in the web browser to force reload the UI page.

2019-08-29 UBA-13020 Anomalies migrated from test-mode to active-mode won't be pushed to ES

Splunk Direct - Cloud Storage does not expose src_ip field

Workaround:
2019-08-06 UBA-12910 When ingesting Office 365 Sharepoint/OneDrive logs through Splunk Direct - Cloud Storage, add an additional field mapping for
be mapped from ClientIP (| eval src_ip=ClientIP). Make sure to add src_ip in the final list of fields selected using the fi
| fields app,change_type,dest_user,file_hash,file_size,object,object_path,object_type,parent_c
_hash,sourcetype,src_user,tag,src_ip

Fixed issues in Splunk UBA

This version of Splunk UBA fixes the following issues. If no issues are listed, none have been reported.

Date
Issue number Description
resolved
2023-02-03 UBA-15961 keyutils-libs = 1.5.10-6.el8 is needed by (installed) keyutils-1.5.10-6.el8.x86_64

UBA-16910,
2023-02-02 UnsupportedOperationException: empty.reduceLeft in BoxPatternModel
UBA-16716

UBA-16911,
2023-02-02 SuspiciousEmailDetectionModel NullPointerException
UBA-16292

Caspida start-all command should check/wait for all the job agents to be available before issuing the
2023-01-30 UBA-16774
command to restart live datasources

Large UBA deployments hit java.lang.OutOfMemoryError due to Hypergraph based Malware Threat
2023-01-26 UBA-15811
Detection Model

2023-01-22 UBA-16831 Entering a UBA anomaly action rule at the UI results in error of "cannot extract elements from a scalar"

2023-01-03 UBA-16492 Hadoop Upgrade creates large backup folders. Need to finalize upgrade.

Assets Cache Update Query Does Not Support Multi-values Data and Causes Postgres Log Size
2022-12-16 UBA-16555
Increase

2022-11-15 UBA-16424 Custom models don't run automatically in 5.1.0.X

9
 Date
Issue number Description
resolved

2022-10-05 UBA-15164 Download Diagnostics "Parsers" for multi-node misses /var/log/caspida/jobexecutor*

2022-09-26 UBA-16310 failed to run: apt-mark hold zookeeper zookeeper-server redis-server redis-tools, exiting

7-node UBA deployment has an invalid value for system.messaging.rawdatatopic.retention.time in

2022-09-09 UBA-16289
caspidatunables-7_node.conf

2022-09-02 UBA-16206 Problem with SSO settings EntityID and Issuer in UI

2022-08-23 UBA-16182 Troubleshooting UI page for the Hadoop Service App does not load

UBA-14287,
2022-08-12 Issue while deleting datasource referencing other UBA original primary cluster
UBA-17142

"Class name not accepted" java.io.InvalidClassException when starting a Splunk connector data source
2022-08-09 UBA-16004
and Splunk defined Source Type

UI will error out when trying to edit the name of an output connector without retyping the password
2022-08-07 UBA-15871
"EVP_DecryptFinal_ex:bad decrypt"

Date resolved Issue number Description

2023-01-19 TEA-483 Spark offline model workflow in UBA 5.1 stops once one model fails

Log4j in Splunk UBA 5.2.0

In Splunk UBA version 5.2.0, all Log4j related jars in the OS packages have either been removed or replaced by Reload4j
besides the following. The following packages have been patched to remove vulnerable classes:

1. Log4j 1.2.x in the Impala OS library /usr/lib/impala/lib (inside the Impala container), also visible in
/var/vcap/store/docker/overlay2/.../usr/lib/impala/lib/:
1. org/apache/log4j/net/SocketAppender.class: CVE-2019-17571
2. org/apache/log4j/net/SocketServer.class: CVE-2019-17571
3. org/apache/log4j/net/SMTPAppender$1.class: CVE-2020-9488
4. org/apache/log4j/net/SMTPAppender.class: CVE-2020-9488
5. org/apache/log4j/net/JMSAppender.class: CVE-2021-4104
6. org/apache/log4j/net/JDBCAppender.class: CVE-2022-23305
7. org/apache/log4j/chainsaw/*.class: CVE-2022-23307
2. Log4j 2.x in the Hive OS library /usr/lib/hive/lib:
1. org/apache/logging/log4j/core/lookup/JndiLookup.class: CVE-2021-44228
3. Non-core Log4j files which do not contain critical or high vulnerabilities.

10
Where to find help

Getting help with Splunk UBA

In addition to the documentation, you can contact Support or Splunk Professional Services for assistance with Splunk
UBA.

• For assistance installing, upgrading, or scaling a Splunk User Behavior Analytics deployment, contact the Splunk
Professional Services team.
• For assistance on a support issue with Splunk User Behavior Analytics, file a case or contact Splunk Support.

11
Third-party software

Third-party credits in Splunk UBA

Splunk User Behavior Analytics contains libraries that were written by others, and are being redistributed as part of Splunk
User Behavior Analytics under their respective open source licenses. We want to thank the contributors to these projects.

This product includes GeoLite2 Data created by MaxMind, available from https://www.maxmind.com.

For complete third-party software information for Splunk User Behavior Analytics, download this separate PDF file:

• Splunk User Behavior Analytics Third-party software credits

12