Intro

During my OSCP studies, I realized I needed a more efficient system for

cracking password hashes. The screaming CPU fans and high CPU usage
became a problem. I first tried using hashcat and the GPU on my MacBook Pro
in OS X. There are some bugs and problems with hashcat on OS X that would
make it crash in the middle of cracking a hash. Also, I was not interested in
investing a server with a bunch of GPUs, the high costs to do this would
outweigh the amount of time I need the system. All of this lead me to do a little
research and found the instructions in the following link to build an AWS
instance for password cracking.

https://medium.com/@iraklis/running-hashcat-v4-0-0-in-amazons-aws-
new-p3-16xlarge-instance-e8fab4541e9b
Since that post was created there have been some changes to the offerings in
AWS EC2 leading me write this post.

If you wish to skip ahead I have created scripts to automate the processes in
the rest of this post. They are both in my github and can be downloaded at the
following links.

https://github.com/suidroot/AWSScripts/blob/master/aws-ec2-create-
kracker.sh
https://github.com/suidroot/AWSScripts/blob/master/configure-kracker.sh
For the rest of the article I will cover some of the instance options in EC2,
installation of the needed Linux packages, the basic setup of Hashcat, running
Hashcat, and finally monitoring and benchmarks of an EC2 instance.

AWS EC2 Options

There are many options for EC2 instances, they have a huge range in cost and
scale.

 G3 – https://aws.amazon.com/ec2/instance-types/g3/
 P2 – https://aws.amazon.com/ec2/instance-types/p2/

I found the g3 instances to be the more cost effective tier. For my testing I opted
to use the g3.4xlarge tier. Next to choose the AMI image, appropriate the
appropriate operating system.

AMI images
There are two options that are I tested hashcat on they are both Ubuntu based.
I’m sure there are many other available options that will work too, but I am
familiar with Ubuntu systems. The first option is a standard Ubuntu image,
there is nothing special about this image and it requires configuration to add the
GPU drivers and a little more work.

Standard Ubuntu
The next option is a Deep Learning image, this image is preconfigured with the
GPU drivers and was originally designed for machine learning applications. I
found the the pre-configuration allowed for me skip a few steps in building out a
new system.

Deep learning Ubuntu GPU driver preloaded

Instance Build and config

Once you have the instance deployed there are a few steps to get the Instance
prepared for hashcat, the steps are a little bit different between a Standard and
a Deep Learning Ubuntu instance.

An apt cronjob may already be running and you will have to wait it out.

Prepare Machine (Standard Ubuntu)

This script will install all the required packages and the Nvidia GPU drivers on a
vanilla Ubuntu installation.

#!/bin/bash

# mostly copied from: https://medium.com/@iraklis/running-hashcat-v4-0-0-in-amazons-aw

#
sudo apt-get update -yq
sudo apt-get install -yq build-essential linux-headers-$(uname -r) unzip p7zip-full lin
sudo apt-get install -yq ocl-icd-libopencl1 opencl-headers clinfo
#sudo apt-get install -yq libhwloc-plugins libhwloc5 libltdl7 libpciaccess0 libpocl2 l
sudo apt-get install -yq python3-pip
pip3 install psutil

sudo touch /etc/modprobe.d/blacklist-nouveau.conf

sudo bash -c "echo 'blacklist nouveau' >> /etc/modprobe.d/blacklist-nouveau.conf"
sudo bash -c "echo 'blacklist lbm-nouveau' >> /etc/modprobe.d/blacklist-nouveau.conf"
sudo bash -c "echo 'options nouveau modeset=0' >> /etc/modprobe.d/blacklist-nouveau.con
sudo bash -c "echo 'alias nouveau off' >> /etc/modprobe.d/blacklist-nouveau.conf"
sudo bash -c "echo 'alias lbm-nouveau off' >> /etc/modprobe.d/blacklist-nouveau.conf"

sudo touch /etc/modprobe.d/nouveau-kms.conf

sudo bash -c "echo 'options nouveau modeset=0' >> /etc/modprobe.d/nouveau-kms.conf"
sudo update-initramfs -u
sudo reboot
### Install nVidia Drivers
wget http://us.download.nvidia.com/tesla/410.104/NVIDIA-Linux-x86_64-410.104.run
sudo /bin/bash NVIDIA-Linux-x86_64-410.104.run --ui=none --no-questions --silent -X

Prepare Machine (Deep Learning Ubuntu)

In comparison the previous script there is a much simpler script to prepare the
Deep Learning instance. The main focus is installing the needed archive
extraction tools.

#!/bin/bash

sudo apt update

sudo apt upgrade
sudo apt install clinfo unzip p7zip-full
sudo apt install build-essential linux-headers-$(uname -r) # Optional
sudo apt-get install -yq python3-pip
pip3 install psutil

Hashcat Setup
Now we need to download and extract the star of the show Hashcat. The link in
the wget below points to the the most recent version as of writing however you
might want to check to see if there is a more recent version at the main site:
https://hashcat.net/hashcat/

wget https://hashcat.net/files/hashcat-5.1.0.7z
7z x hashcat-5.1.0.7z

Download wordlists

You will need some wordlists for hashcat to use to crack passwords, he
commands listed are for some wordlists I like to use when cracking. You should
however add whichever lists are your favories.

mkdir ~/wordlists
git clone https://github.com/danielmiessler/SecLists.git ~/wordlists/seclists
wget -nH http://downloads.skullsecurity.org/passwords/rockyou.txt.bz2 -O ~/wordlists/r
cd ~/wordlists
bunzip2 ./rockyou.txt.bz2
cd ~

Running hashcat
Now it is time to run hashcat and crack some passwords. When running hashcat
I had the best performance with the arguments-O -w 3. Below is an example
command line I’ve used inclusing a rules file.
./hashcat-5.1.0/hashcat64.bin --username -m 1800
./megashadow256.txt wordlists/rockyou.txt -r hashcat-
5.1.0/rules/best64.rule -O -w 3

Monitoring the Nvidia GPU

The nvidia-smi utility can be used to show the GPU processor usage and what
processes are utilizing the GPU(s). The first example is is showing an idle GPU.

ubuntu@ip-172-31-17-6:~$ sudo nvidia-smi

Fri Apr 26 14:43:49 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.104 Driver Version: 410.104 CUDA Version: 10.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla M60 Off | 00000000:00:1E.0 Off | 0 |
| N/A 37C P0 42W / 150W | 0MiB / 7618MiB | 97% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+
This example shows a GPU being used by hashcat.

ubuntu@ip-172-31-17-6:~$ sudo nvidia-smi

Fri Apr 26 14:44:44 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.104 Driver Version: 410.104 CUDA Version: 10.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla M60 Off | 00000000:00:1E.0 Off | 0 |
| N/A 46C P0 141W / 150W | 828MiB / 7618MiB | 100% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 11739 C ./hashcat-5.1.0/hashcat64.bin 817MiB |
+-----------------------------------------------------------------------------+

Conclusion and Benchmarks

Finally here is a benchmark I ran on a g3.4xlarge instance. This instance type
contains 1 GPU. These results give an idea of performance for this AWS EC2
instance type.
ubuntu@ip-172-31-17-6:~$ ./hashcat-5.1.0/hashcat64.bin -O -w 3 -b
hashcat (v5.1.0) starting in benchmark mode...

* Device #2: Not a native Intel OpenCL runtime. Expect massive speed loss.
You can use --force to override, but do not report related errors.
nvmlDeviceGetFanSpeed(): Not Supported

OpenCL Platform #1: NVIDIA Corporation

======================================
* Device #1: Tesla M60, 1904/7618 MB allocatable, 16MCU

OpenCL Platform #2: The pocl project

====================================
* Device #2: pthread-Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz, skipped.

Benchmark relevant options:

===========================
* --optimized-kernel-enable
* --workload-profile=3

Hashmode: 0 - MD5

Speed.#1.........: 11611.6 MH/s (90.74ms) @ Accel:512 Loops:512 Thr:256 Vec:4

Hashmode: 100 - SHA1

Speed.#1.........: 4050.2 MH/s (65.01ms) @ Accel:512 Loops:128 Thr:256 Vec:2

Hashmode: 1400 - SHA2-256

Speed.#1.........: 1444.5 MH/s (91.98ms) @ Accel:256 Loops:128 Thr:256 Vec:1

Hashmode: 1700 - SHA2-512

Speed.#1.........: 499.4 MH/s (66.78ms) @ Accel:128 Loops:64 Thr:256 Vec:1

Hashmode: 2500 - WPA-EAPOL-PBKDF2 (Iterations: 4096)

Speed.#1.........: 189.8 kH/s (42.76ms) @ Accel:128 Loops:64 Thr:256 Vec:1

Hashmode: 1000 - NTLM

Speed.#1.........: 18678.1 MH/s (56.58ms) @ Accel:512 Loops:512 Thr:256 Vec:2

Hashmode: 3000 - LM

Speed.#1.........: 10529.6 MH/s (50.60ms) @ Accel:128 Loops:1024 Thr:256 Vec:1

Hashmode: 5500 - NetNTLMv1 / NetNTLMv1+ESS

Speed.#1.........: 10650.8 MH/s (49.60ms) @ Accel:512 Loops:256 Thr:256 Vec:1

Hashmode: 5600 - NetNTLMv2

Speed.#1.........: 829.3 MH/s (80.24ms) @ Accel:256 Loops:64 Thr:256 Vec:1

Hashmode: 1500 - descrypt, DES (Unix), Traditional DES

Speed.#1.........: 442.0 MH/s (37.81ms) @ Accel:4 Loops:1024 Thr:256 Vec:1

Hashmode: 500 - md5crypt, MD5 (Unix), Cisco-IOS $1$ (MD5) (Iterations: 1000)

Speed.#1.........: 4209.1 kH/s (51.39ms) @ Accel:1024 Loops:500 Thr:32 Vec:1

Hashmode: 3200 - bcrypt $2*$, Blowfish (Unix) (Iterations: 32)

Speed.#1.........: 7572 H/s (33.02ms) @ Accel:16 Loops:4 Thr:8 Vec:1

Hashmode: 1800 - sha512crypt $6$, SHA512 (Unix) (Iterations: 5000)

Speed.#1.........: 76958 H/s (83.99ms) @ Accel:512 Loops:128 Thr:32 Vec:1

Hashmode: 7500 - Kerberos 5 AS-REQ Pre-Auth etype 23

Speed.#1.........: 149.4 MH/s (56.00ms) @ Accel:128 Loops:64 Thr:64 Vec:1

Hashmode: 13100 - Kerberos 5 TGS-REP etype 23

Speed.#1.........: 152.1 MH/s (55.00ms) @ Accel:128 Loops:64 Thr:64 Vec:1

Hashmode: 15300 - DPAPI masterkey file v1 (Iterations: 23999)

Speed.#1.........: 32703 H/s (84.02ms) @ Accel:256 Loops:64 Thr:256 Vec:1

Hashmode: 15900 - DPAPI masterkey file v2 (Iterations: 7999)

Speed.#1.........: 21692 H/s (96.24ms) @ Accel:256 Loops:128 Thr:32 Vec:1

Hashmode: 7100 - macOS v10.8+ (PBKDF2-SHA512) (Iterations: 35000)

Speed.#1.........: 5940 H/s (40.09ms) @ Accel:64 Loops:32 Thr:256 Vec:1

Hashmode: 11600 - 7-Zip (Iterations: 524288)

Speed.#1.........: 4522 H/s (55.87ms) @ Accel:256 Loops:128 Thr:256 Vec:1

Hashmode: 12500 - RAR3-hp (Iterations: 262144)

Speed.#1.........: 18001 H/s (56.74ms) @ Accel:4 Loops:16384 Thr:256 Vec:1

Hashmode: 13000 - RAR5 (Iterations: 32767)

Speed.#1.........: 18135 H/s (55.93ms) @ Accel:128 Loops:64 Thr:256 Vec:1

Hashmode: 6211 - TrueCrypt PBKDF2-HMAC-RIPEMD160 + XTS 512 bit (Iterations: 2000)

Speed.#1.........: 121.7 kH/s (59.39ms) @ Accel:128 Loops:32 Thr:256 Vec:1

Hashmode: 13400 - KeePass 1 (AES/Twofish) and KeePass 2 (AES) (Iterations: 6000)

Speed.#1.........: 68380 H/s (158.89ms) @ Accel:512 Loops:256 Thr:32 Vec:1

Hashmode: 6800 - LastPass + LastPass sniffed (Iterations: 500)

Speed.#1.........: 1088.7 kH/s (48.51ms) @ Accel:128 Loops:62 Thr:256 Vec:1

Hashmode: 11300 - Bitcoin/Litecoin wallet.dat (Iterations: 199999)

Speed.#1.........: 2107 H/s (78.97ms) @ Accel:128 Loops:64 Thr:256 Vec:1