Exploiting

Out-of-Order-Execution
Processor Side Channels to Enable
Cross VM Code Execution

Sophia D’Antoine
REcon 2015

The Cloud

06/19/2015 Exploiting Out-of-Order-Execution 2/46

Cloud Computing (IaaS)
• Virtual instances
• Hypervisors

Dynamic allocation
=> Reduces cost

06/19/2015 Exploiting Out-of-Order-Execution 3/46

Everyone’s Happy

06/19/2015 Exploiting Out-of-Order-Execution 4/46

Problems with the Cloud Security issues with cloud computing • Sensitive data stored remotely • Vulnerable host • Untrusted host • Co-located with foreign VM’s 06/19/2015 Exploiting Out-of-Order-Execution 5/46 .

wat 06/19/2015 Exploiting Out-of-Order-Execution 6/46 . Physical co-location leads to side channel vulnerabilities.

Cloud Hardware 06/19/2015 Exploiting Out-of-Order-Execution 7/46 .

Universal Vulnerabilities 1) Translation between physical and virtual hardware based on need 2) Allocation causes contention 3) Private VM activities not opaque to aaco-residents 06/19/2015 Exploiting Out-of-Order-Execution 8/46 .

Targeting the processor 4. Importance of memory models 5. Introduction 2.Overview 1. Cloud exploitation techniques 3. Demo 7. Conclusion 06/19/2015 Exploiting Out-of-Order-Execution 9/46 . Design of an Out-of-Order-Execution channel 6.

Side Channel Attack “In cryptography. a side. Cloud Computing channel attack is any attack • Hardware side based on information gained channel from the physical • Cross virtual implementation of a machine cryptosystem” • Information gained through recordable changes in the system 06/19/2015 Exploiting Out-of-Order-Execution 10/46 .

Classification S/R Model • Hardware agnostic • Two methods of interacting – Transmit – Receive transmit: force receive: record artifacts artifacts Hardware 06/19/2015 Exploiting Out-of-Order-Execution 11/46 .

communication (C&C) 06/19/2015 Exploiting Out-of-Order-Execution 12/46 . DoS 2. broadcast signal • Transmit (infiltrate) 1. crypto key theft 2. co-residency • Transmit & Receive (network) 1. process monitoring 3. Possible Exploits • Receive (exfiltrate) 1. environment keying 4.

Communication Virtual VM1 VM2 Client Master VM Allocations S R S R S R Shared Communication Medium Hardware 06/19/2015 Exploiting Out-of-Order-Execution 13/46 .

pdf 06/19/2015 Exploiting Out-of-Order-Execution 14/46 .re/cache.Cache Side Channel Example [3] Flush+Reload targets the L3 Cache Tier • Receiving Mechanism (Adversary) – Flushes & queries • Transmitting Mechanism (Victim) – Accesses same L3 line • Leaked GnuPG Private Key sophia.

etc.Pipeline vs Cache Channel Benefits: • Quiet. • Channel & noise amplifies in a crowded cloud environment 06/19/2015 Exploiting Out-of-Order-Execution 15/46 . covert channel • Not affected by cache misses.

Importance of memory models 5. Demo 7. Conclusion 06/19/2015 Exploiting Out-of-Order-Execution 16/46 . Targeting the pipeline 4. Design of an Out-of-Order-Execution channel 6. Introduction 2. Cloud exploitation techniques 3.Overview 1.

The Attack Vector Side Channels which Exploit Hardware Vulnerabilities Inherent to Modern Cloud Computing Systems Requirements: • Shared hardware • Dynamically allocated hardware resources • Co-Location with adversarial VMs or infected VMs 06/19/2015 Exploiting Out-of-Order-Execution 17/46 .

=> CPU’s pipeline => System artifacts queried dynamically • Instruction order • Results from instruction sets 06/19/2015 Exploiting Out-of-Order-Execution 18/46 .Pipeline Side Channel We chose to target the processor as the hardware medium.

Out-of-Order-Execution 06/19/2015 Exploiting Out-of-Order-Execution 19/46 .

Processor Pipeline Contention VM VM VM VM Process01 Process02 Process03 Process04 SMT Pipeline Optimizes Core01 Core02 Executing Shared Instructions Hardware Processor From Foreign Applications 06/19/2015 Exploiting Out-of-Order-Execution 20/46 .

RECEIVER 06/19/2015 Exploiting Out-of-Order-Execution 21/46 .

Record Out of Order Execution [6] 06/19/2015 Exploiting Out-of-Order-Execution 22/46 .

[Y] load r2. [X] Order store [X].Record Out of Order Execution THREAD 1 THREAD 2 => Synched store [X]. 1 store [Y]. [X] r1 = r2 = 1 => Asynched store [X]. [Y] store [Y]. 1 load r1. 1 r1 = r2 = 0 Execution 06/19/2015 Exploiting Out-of-Order-Execution 23/46 . 1 store [Y]. 1 load r2. [Y] load r2. 1 load r1. [X] r1 = 0 r2 = 1 => Out of load r1.

. Averages matter sem_wait(endSema1 & endSema2).Record Out of Order Execution int X. for (int iterations = 1.Y. NULL). NULL. sem_post(beginSema1 & beginSema2).initialize semaphores Sema1 & Sema2… pthread_t thread1.Y = 0. pthread_create(&threadN. threadNFunc. if (r1 == 0 && r2 == 0) count_OoOE ++. 06/19/2015 Exploiting Out-of-Order-Execution 24/46 . iterations++) X. thread2. ….count_OoOE.

TRANSMITTER 06/19/2015 Exploiting Out-of-Order-Execution 25/46 .

Force Out of Order Execution Mfence: ● x86 instruction full memory barrier prevents memory reordering of any kind ● order of 100 cycles per operation ● … mov dword ptr [_spin1]. 0 … mfence … mov dword ptr [_spin2]. 0 … mfence 06/19/2015 Exploiting Out-of-Order-Execution 26/46 .

Exploiting Out-of-Order-Execution 27/46 .Force Out of Order Execution THE PIPELINE …. NOP Store [X]... 1 mfence Load r1. [X] NOP ….

Demo 7. Cloud exploitation techniques 3. Design of an Out-of-Order-Execution channel 6. Introduction 2. Importance of memory models 5.Overview 1. Conclusion 06/19/2015 Exploiting Out-of-Order-Execution 28/46 . Targeting the processor 4.

Types of Memory Reordering Memory Reordering Processor Compilation (Run) Time Time GCC OoOE Execution Multithreaded MultiCored Programs (MultiExecution Processors) Computers 06/19/2015 Exploiting Out-of-Order-Execution 29/46 .

Types of Memory Reordering Dynamic side channel artifacts Processor (Run) Time OoOE Execution MultiCored (MultiExecution Processors) Computers 06/19/2015 Exploiting Out-of-Order-Execution 30/46 .

[7] 06/19/2015 Exploiting Out-of-Order-Execution 31/46 .

Instruction A visible to all processes before B occurs .Types of Memory Reordering [4. 5] .#StoreLoad most expensive operation 06/19/2015 Exploiting Out-of-Order-Execution 32/46 .

Force Out of Order Execution Memory Barrier ● ‘Lock-free programming’ on SMT multiprocessors ● #StoreLoad unique prevents r1=r2=0 ● x86: mfence ( effects the pipeline ) 06/19/2015 Exploiting Out-of-Order-Execution 33/46 .

• Out-of-Order-Execution • 06/19/2015 Exploiting Out-of-Order-Execution 34/46 .

Overview 1. Importance of memory models 5. Conclusion 06/19/2015 Exploiting Out-of-Order-Execution 35/46 . Targeting the processor 4. Design of an Out of Order Execution channel 6. Demo 7. Introduction 2. Cloud exploitation techniques 3.

Lab Model Scheduler Xen hypervisor ● Popular commercial IaaS platforms Xeon Processors Shared multi-core/ multi-processor hardware ● 8 logical CPU’s/ 4 cores ● 6 virtual machines (VM’s) ● Parallel Processing/ Simultaneous Multi-Threading On (SMT) 06/19/2015 Exploiting Out-of-Order-Execution 36/46 .

Virtual Machines • 6 Windows 7 VM’s VM1 VM2 VM3 VM4 VM5 VM6 P1 P2 P3 P4 CPU1 CPU1 06/19/2015 Exploiting Out-of-Order-Execution 37/46 .

Virtual Machine S/R 06/19/2015 Exploiting Out-of-Order-Execution 38/46 .

Cloud exploitation techniques 3. Introduction 2. Targeting the processor 4. Demo 7. Design of an Out-of-Order-Execution channel 6. Importance of memory models 5.Overview 1. Conclusion 06/19/2015 Exploiting Out-of-Order-Execution 39/46 .

py 06/19/2015 Exploiting Out-of-Order-Execution 40/46 .Demo Links sophia.re/receiver.re/sender.py sophia.

Overview 1. Conclusion 06/19/2015 Exploiting Out-of-Order-Execution 41/46 . Demo 7. Importance of memory models 5. Introduction 2. Design of an Out-of-Order-Execution channel 6. Cloud exploitation techniques 3. Targeting the processor 4.

Potential Channel Mitigation Protected Resource Ownership ● Isolating VM’s ● Turn off hyperthreading ● Blacklisting resources for concurrent threads ● Downside: cloud benefits 06/19/2015 Exploiting Out-of-Order-Execution 42/46 .

Contribution: We demonstrate a novel Out of Order Execution side channel..In Conclusion. • Dynamic querying/ forcing method • Application to cloud computing • Mitigation techniques 06/19/2015 Exploiting Out-of-Order-Execution 43/46 ..

Trail of Bits 06/19/2015 Exploiting Out-of-Order-Execution 44/46 .Acknowledgements .RPISEC .Jeremy Blackthorne .

pdf 06/19/2015 Exploiting Out-of-Order-Execution 45/46 .re/thesis. #pwning) email: sophia@trailofbits.Any Questions? IRC: quend (#rpisec.com thesis link: sophia.

usenix.com/2008/11/05/who-ordered-memory-fences-on-an-x86/ [5] http://preshing.References [1] http://www.com/Assets/en_US/PDF/manual/253668.com/web-hosting-news/aws-to-reach-24-billion-in-revenue-by-2022-morgan-stanley [2] http://www.com/20120710/memory-barriers-are-like-source-control-operations/ 06/19/2015 Exploiting Out-of-Order-Execution 46/46 .thewhir.com/sites/louiscolumbus/2015/01/24/roundup-of-cloud-computing-forecasts-and-market- estimates-2015/ [3] https://www.pdf [7] http://preshing.pdf [4] http://bartoszmilewski.org/system/files/conference/usenixsecurity14/sec14-paper-yarom.forbes.com/20120930/weak-vs-strong-memory-models/ [8] http://en.org/wiki/Memory_barrier#An_illustrative_example [9] http://preshing.com/20120913/acquire-and-release-semantics/ [6] http://www.intel.wikipedia.