Concurrency with Multiprocessing in Python

PhillyPug & Philly.rb RedSnake Meeting

February 8th, 2011

Gavin M. Roy
myYearbook.com
green threads in threading
from threading import Thread
import time

class MyThread(Thread):

def run(self):
print "%s running" % self.name
time.sleep(5)
print "%s done" % self.name

for x in xrange(0,10):
thread = MyThread()
thread.start()
thread.join(0)
gmr-0x04:pika gmr$ python threads.py
MyThread-1 running
MyThread-2 running
MyThread-3 running
MyThread-4 running
MyThread-5 running
MyThread-6 running
MyThread-7 running
MyThread-8 running
MyThread-9 running
MyThread-10 running
MyThread-1 done
MyThread-2 done
MyThread-3 done
MyThread-4 done
MyThread-5 done
MyThread-6 done
MyThread-7 done
MyThread-8 done
MyThread-9 done
MyThread-10 done
threading

• Locks

• Reentrant Locks
Reentrant knows who
own the locks and the
recursion level
• Conditions
Additional classes like
Queue.Queue

• Semaphores

• Events

• Timers
This GIL From David Beeazley’s GIL Visualization
http://www.dabeaz.com/GIL/gilvis/fourthread.html
enter multiprocessing
multiprocessing module

• All the things threading has • Connections

• Exchanging Objects • Managers TCP Server coordinating

shared objects

• Queues and Pipes • SyncManager Process Sync

• Shared State • Logging

• Pipes and Queues

• Pools
from multiprocessing import Process
import time

class MyThread(Process):

def run(self):
print "%s running" % self.name
time.sleep(5)
print "%s done" % self.name

for x in xrange(0,10):
thread = MyThread()
thread.start()
thread.join(0)
gmr-0x04:pika gmr$ python processes.py
Thread-1 running
Thread-2 running
Thread-3 running
Thread-4 running
Thread-5 running
Thread-6 running
Thread-7 running
Thread-8 running
Thread-9 running
Thread-10 running
Thread-1 done
Thread-2 done
Thread-3 done
Thread-4 done
Thread-5 done
Thread-6 done
Thread-7 done
Thread-8 done
Thread-9 done
Thread-10 done
multiprocessing.reduction
Photo By Susan NYC: http://www.flickr.com/photos/en321/33868864/
# Process #1
from multiprocessing.reduction import reduce_handle
import socket

# Create a socket
sock = socket.socket(socket.AF_INET,
socket.SOCK_STREAM, 0)

# Do stuff here, client or server wise

# Create the pickled socket handle

handle = reduce_handle(sock.fileno)

# Process #2
from multiprocessing.reduction import rebuild_handle
import socket

# In other process
fd = rebuild_handle(handle)
sock = socket.fromfd(fd, socket.AF_INET,
socket.SOCK_STREAM)

# Now I can read and write from the socket too