Professional Documents
Culture Documents
v1.7
18 JUL 2022
Introduction ........................................................................................................................... 5
Intended Audience............................................................................................................ 5
Reaching the support ........................................................................................................ 5
High level information gathering .................................................................................... 5
Agent ..................................................................................................................................... 6
Supported Versions .......................................................................................................... 6
Windows ........................................................................................................................... 7
Policies............................................................................................................................ 7
Priorities Schema ........................................................................................................ 7
Whitelist not working ................................................................................................. 8
Assessing an AppDir path........................................................................................... 8
Protection Policy did not work ................................................................................... 8
Antimalware ................................................................................................................... 9
Network access issues................................................................................................ 9
Slowness related to the Anti-Malware ....................................................................... 9
Generalized slowness ............................................................................................. 9
Installation .................................................................................................................... 10
Citrix VDI Deployment ............................................................................................. 10
Installer log collection ............................................................................................... 11
Network Resource Unavailable ................................................................................ 11
To resolve this issue and proceed with the installation, .......................................... 11
1.Start regedit.exe with administrator privilege
.................................................................................................................................. 11
Registration Errors.................................................................................................... 12
Re-registration .......................................................................................................... 13
Registration Reachability test ................................................................................... 14
Agent Manual removal ............................................................................................. 15
Performance ................................................................................................................. 16
Performances information collection through xperf ................................................ 16
Crash dumps ................................................................................................................ 17
2 of 28
Keeper crash / Dump collection .............................................................................. 17
Handling a BSOD Loop ............................................................................................ 17
Other ............................................................................................................................ 18
Forensics Kit fails...................................................................................................... 18
Linux ................................................................................................................................ 18
Installation .................................................................................................................... 18
Uninstallation/Manual Removal ................................................................................... 18
Registration Errors ....................................................................................................... 18
Performances ............................................................................................................... 19
Other ............................................................................................................................ 20
Collect troubleshooting logs from Linux .................................................................. 20
MacOS ............................................................................................................................. 20
Installation .................................................................................................................... 20
Uninstallation/Manual Removal ................................................................................... 20
Registration Errors ....................................................................................................... 21
Performances ............................................................................................................... 22
Other ............................................................................................................................ 22
Collect log errors: ..................................................................................................... 22
Backend ............................................................................................................................... 23
General Maintenance...................................................................................................... 23
Graceful shutdown....................................................................................................... 23
Full Server Shutdown ............................................................................................... 23
ReaQta Services Shutdown ...................................................................................... 23
Server startup............................................................................................................... 23
Verify License Validity .................................................................................................. 23
Increasing Disk/RAM size ............................................................................................ 23
Essential log collection in a closed environment (On-Premise / Air Gapped) ............. 23
Frontend/UI ......................................................................................................................... 24
Email configuration ......................................................................................................... 24
Invalid OTP (On-Premise / Airgapped environments) .................................................. 25
CyberAssistant Troubleshooting.................................................................................... 25
CyberAssistant not closing/rescoring/creating whitelist policies ................................ 25
CyberAssistant configuration looks correct but the alert has not being handled ....... 25
3 of 28
CyberAssistant showing “Something went wrong” for each alert ............................... 26
Support Request Form ........................................................................................................ 26
Support Request Form.................................................................................................... 28
4 of 28
Introduction
The scope of the Support Guidelines is to offer the necessary steps to classify, collect
information and solve non-bug related issues.
Intended Audience
The present document is intended for the IT staff of partners and customers responsible
for the solution.
The Partners should handle the L1 (First Line) of support, these guidelines cover the entire
L1 with the aim to:
1. Solve the issue in autonomy
2. If the issue cannot be solved by the L1, the guideline contains the steps to collect
the necessary information to be forwarded to the ReaQta Support.
Starting from 1st of September 2022 the address will change in Reaqta.Support@ibm.com
Please ensure that every support request follows the High-level information as described
in detail in the section below.
Data collection should always be contextualized. Please attach the following information
to the data collected during a support case:
5 of 28
5. Issue surface: Is the issue impacting one or more endpoints?
6. Reproducible: yes/no
7. Version information: collect all the versions (backend/frontend/agent) involved
into the issue
8. Known issue: yes/no
Agent
Supported Versions
IMPORTANT: From 3.9 onwards, only TLS 1.2 is supported for communication with the
server. Please make sure your endpoint is configured to use TLS 1.2. See here for more
information: https://support.microsoft.com/en-us/topic/update-to-enable-tls-1-1-and-tls-
1-2-as-default-secure-protocols-in-winhttp-in-windows-c4bd73d2-31d7-761e-0178-
11268bb10392
6 of 28
Windows
Policies
Priorities Schema
The Priorities Schema can be found in the “Policies” by hovering on one of the policies; a
checkbox will appear on the very first column before the “Type” one. Once selected, a
blue bar will appear on the bottom, by clicking on “Show Priority” in the above schema.
The schema is divided into 3 columns where the priority has the following order (from
higher to lower): Scope, Type, Matcher.
Each category enumerates from the higher to the lower priority, top to bottom, such as:
7 of 28
Whitelist not working
A not working whitelist means that, after creating it, you are still getting the corresponding
alert. The first step is to assess if the policy is correct:
If the above are correct, the next step is to verify the priorities:
● If the alert is the result of a Protection => Check the scope of the WL and the
Group or in the MSSP case, Client/Subgroup of the Endpoint respects the Priority
Schema above.
8 of 28
5. If the policy is loaded and the alert happened after the reception, contact the
Support
Antimalware
Check if the issue is solved and contact in any case the Support to inform them about the
created policy.
Generalized slowness
9 of 28
3. From the threat hunt, search for custom event no process and identify the
“timeout” events:
Collect the full path of the application, that might be in the DOS-Style path, an example is
shown below:
\Device\HarddiskVolume5\Users\USERI\AppData\Local\Mozilla\Firefox\Profiles\yrfv8n4
g.default-release\cache2\entries\E90A8AB86882E9EFCD54ABE91EB3876FA017D1EC
C:\Users\*\AppData\Local\Mozilla\Firefox\Profiles\yrfv8n4g.default-
release\cache2\entries\*
Verify if the performance improves, if not, collect the xperf data by following the
instructions given in the performance paragraph.
Installation
10 of 28
Installer log collection
In situations where the installation fails for errors not related to the registration, it’s
necessary to collect the installation logs as described below:
11 of 28
Registration Errors
The registration error log is located into the folder %TEMP% (the expanded path is the
following: C:\Users\<Username>\AppData\Local\Temp )and begins with rqt_installer
12 of 28
409 The endpoint is already registered, in the
endpointId field you can find the endpoint already
registered. Usually, it’s associated with a cloned
machine, in that case it is necessary to perform a
sysprep.
Re-registration
In presence of an error message like the following one:
It means that the machine experienced a hardware change which invalidated the local
license id. It’s currently necessary to:
13 of 28
● From the endpoint, uninstall the agent from the command line
● Reinstall as usual
● Try to reach from the browser, on the endpoint having issues the given registration
address
The expected and correct response (server working and endpoint able to reach it) is the
following:
Once the above are verified if the problem still persists, it's suggested to check the
network settings of the infrastructure:
Attempt, if possible, to use a different network connection (i.e. Tethering / Mobile hotspot) to
register the endpoint, if the registration succeeds, the local network setting of the infrastructure
must be reviewed.
14 of 28
Missing Installation Log
In a scenario where there is no installation log, start regedit.exe with admin privilege and
search for anything that is related to ReaQta.
● keeper
● i00
● rqtsentry
● rqtnetsentry
Except for your Antivirus exclusion list, or any other registry key that you deem required,
delete all the entries. Try installing ReaQta-Hive agent again after you have removed the
registry keys.
1. sc stop keeper
2. sc delete keeper
3. sc stop rqtsentry
4. sc delete rqtsentry
5. sc stop rqtnetsentry
6. sc delete rqtnetsentry
7. sc stop i00
8. sc delete i00
9. Delete c:\Program Files\ReaQta
10. Delete c:\windows\system32\drivers\rqtsentry.sys
11. Delete c:\windows\system32\drivers\rqtnetsentry.sys
12. Delete c:\windows\system32\drivers\i00.sys
13. regedit as admin:
14. Search for reaqta in HKLM and remove all the entries
15 of 28
Performance
In order to classify the performance issue, assess which category is:
Application related issue -> A specific application or group of applications is slow in
performance.
System wide -> Performance degradation.
When the performance issue is associated to a specific application:
1. Identify the actions performed by the application and the corresponding executable
full path.
2. Disable the Anti-Ransomware from the “Live Response”. (command:
antiransomware off).
3. Verify if the issue is solved.
4. If the issue is solved, create a whitelist “ransomware behavior” for the given
application.
5. Enable the Anti-Ransomware again, assess if the issue is definitively solved.
(command: antiransomware on).
6. If the issue is not solved, verify from the threat hunt the presence of additional
processes, potentially involved in the same time-frame and create additional
ransomware behavior based whitelists.
7. If the issue persists, disable keeper (cmd.exe as admin: sc stop keeper )
8. Try again and note if the performances change
9. Disable rqtsentry (cmd.exe as admin: sc stop keeper )
10. try again and annotate if the performances change. Enable again by following the
commands: sc start rqtsentry - sc start keeper .
11. Annotate date and time, endpoint name, full application path and contact ReaQta’s
support.
xperf -on
PROC_THREAD+LOADER+PROFILE+FLT_IO_INIT+FLT_IO+FLT_FASTIO+FLT_IO_FAILURE+F
ILENAME+DISK_IO+DISK_IO_INIT -stackwalk
Profile+MiniFilterPreOpInit+MiniFilterPostOpInit+DiskReadInit+DiskWriteInit+DiskFlushInit -
BufferSize 1024 -MinBuffers 256 -MaxBuffers 256 -MaxFile 256 -FileMode Circular
Wait for 5 mins. If a specific application is noticeably slow, open and use it during the
xperf capture time.
xperf -stop -d minifilter_and_diskio.etl
16 of 28
Collect memory logs by using:
https://github.com/zodiacon/AllTools/blob/master/PoolMonXv2.exe
Open the application as Administrator and allow it to collect 15mins. of trace logs, save
and send to Support.
Crash dumps
BSOD
In the presence of a BSOD collect the following information:
1. Dashboard / Client name if it’s a MSSP.
2. Endpoint(s).
3. Check if the BSOD is reproducible and provide the steps to reproduce, and
eventually the versions of the software involved.
4. Provide time and date of when it happened.
On the endpoint Check the presence of crash dumps in the following directories:
● C:\Windows\Minidump
● C:\Windows\MEMORY.DMP
17 of 28
Other
Forensics Kit fails
In the presence of a failure in generating the Forensics:
1. On Hover with the mouse on the “Failed” icon available in the column “Status”
2. Collect the error string
3. If it’s: “Package generation failed: Lunch script failed: 669” means that the
PowerShell execution is blocked at endpoint level. Contact the IT administrator to
allow it and try again.
4. If the error is different, contact the Support.
Linux
Installation
Currently, the Linux agent (0.50.0 and older) requires to be installed by IP Address only,
domain resolution is not supported.
Uninstallation/Manual Removal
To uninstall and/or manually remove a Linux agent:
Registration Errors
18 of 28
5. see the Registration Reachability Test
section
6. Start cmd with admin privilege. Run the
following commands to determine if there is a
winhttp proxy
#Show proxy
netsh winhttp show proxy
#Delete proxy
netsh winhttp reset proxy
#Configure proxy
netsh winhttp set proxy <proxy>:<port>
Performances
19 of 28
● Is the performance degradation constant or it’s spot one?
● Describe in detail if there are specific processes/applications/aspect impacted
● If it’s not constant, can you describe what happens and provide a time frame of
when it happened
Other
MacOS
Installation
In case of error during the installation:
Uninstallation/Manual Removal
1. Remove Endpoint from dashboard
2. Go in the following directory: /Library/ReaQta-Hive
3. As root run the following script: uninstall.sh
4. Exit from the directory (ie. cd /)
5. Verify if /Library/ReaQta-Hive is effectively removed
20 of 28
Registration Errors
The error log is visible from the terminal at install time. Once the issue is solved, follow
the “Uninstall/Manual Removal” steps and try again.
21 of 28
442 invalid-license-error-during- the GIDS parameter is missing, it’s mandatory in an
into-group-registration MSSP installation. (See Re-Registration)
Performances
Other
22 of 28
Backend
General Maintenance
In case of a scheduled maintenance to the hosting server and/or every activity that
involves switching off the server hosting the ReaQta brain it’s necessary to gracefully
shutdown the system as described below.
Graceful shutdown
Before powering off a server it is necessary to gracefully shutdown the ReaQta services.
Server startup
As root issue: systemctl start reaqta.service
23 of 28
Follow the below mentioned procedure:
sudo systemctl stop reaqta
sudo fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --
filename=/data/fio-test --bs=4k --iodepth=64 --size=4G --readwrite=randrw
Frontend/UI
Email configuration
SMTP settings are configured but email is not received:
If the above steps do not resolve the issue, collect the following:
● Complete SMTP configuration as the fields in the ReaQta’s dashboard, credentials
can be redacted.
● Server access via SSH and issue the following:
Open repl:
yarn repl
Once repl is loaded, print config:
Configuration.findMailServer().then(c => console.log(c.content))
Send the output to ReaQta’s support.
24 of 28
Invalid OTP (On-Premise / Airgapped environments)
If the Dashboard users have the Two Factor Authentication enabled and at Login time, all
of them are unable to login due to an “Invalid OTP” code, the server administrator is required
to check:
CyberAssistant Troubleshooting
By taking into account the specific Alert on which it did not work, take note of:
• Trigger
• Presence of the protection event
Check if the CyberAssistant configuration matches from Administration > Cyber Assistant and verify
additionally if the trigger belongs to the ones currently handled by consulting on the built -in help in
the top right corner by clicking on: “Read how each configuration works”.
CyberAssistant configuration looks correct but the alert has not being handled
• There are less that 3 nodes in the behavioral tree, in that case the CyberAssistant does not
work
• There are too many nodes in the behavioral tree, in that case the CyberAssistant does not
work.
If the above are with the range, check the presence of errors within the Alert details page
in the section “Cyber Assistant” and report them to Support.
25 of 28
By accessing through SSH the server, verify through a docker ps the presence of the
following containers names running:
1. graphy
2. graphy-gui
3. policycompiler
To collect logs issue the command: docker logs <container_name> > container_name.log
IMPORTANT: Please note that lack of details and context will cause delay in the
investigation.
1. Link to the Dashboard (issue, endpoint etc). If more than one link, use comma ',' to
separate the URLs
5. What was the user doing when the issue was noticed?
7. Link to any files that may help in the troubleshooting e.g. crash dump, screenshots
etc.
1. Link to the Server that best illustrates the issue encountered. If there is more than one
link, use the comma ',' to separate the URLs.
4. Link to any files that may help in the troubleshooting e.g. screenshots etc.
28 of 28