yLhon (2.

3) vlrLual Machlne
A gulded Lour
eLer 1rger
Aprll 2008
yLhon
SLarLed uec 1989 by Culdo van 8ossum
llrsL publlc release leb 1991 on uSLnL1
lnLended as scrlpung language for Amoeba
8enevolenL ulcLaLor lor Llfe (8ulL)
Coogle, Zope, lone, lxar, Mallman,
8lL1orrenL, ?ahoo, nASA, .
yLhon LnhancemenL roposal (L)
lree soware, "ascal of Lhe 2000s"
8eneLs
Comblnauon of good ldeas from oLher
languages - C, C++, Modula, A8C, lcon, .
lnLerpreLer approach, no expllclL Lyplng,
memory managemenL, rsL-class ob[ecL
model, porLablllLy of source and byLe code,
ln-bulld module documenLauon
"8auerles lncluded"
lauorm-neuLral sysLem calls, bullL-ln sLandard
llbrary (neLwork, les, Cul, daLabases, .)
lnLeracuve mode vs. scrlpL mode
yLhon Language
Meanwhlle focus on sLablllLy of language
LvL8?1PlnC ls an ob[ecL
varlables, consLanLs, Lypes, funcuons, .
LvL8?1PlnC has a namespace
LvL8?1PlnC can be prlnLed and lnLrospecLed
uynamlc Lyplng, sLauc scoplng
owerful bullL-ln daLa Lypes: sLrlngs, numerlcal
Lypes, llsLs, dlcuonarles, boolean, .
Speclalues: >>>, self, global, .
yLhon Language - SLrucLure
uenluons and sLaLemenLs can be comblned
Lo an lmporLable !"#$%&
uoued module names for ()*+),& concepL
lmporL LesL / lrom LesL lmporL x / lrom LesL
lmporL * / lmporL LesL as Lhe1esL
lndenLauon Lo show block sLrucLure
Classes
def [methodName] (self, ):
ConsLrucLor denoLed by __init__ meLhod
global sLaLemenL
yLhon Language
ConLrol sLrucLures (if, while, ...)
Collecuon lLeraLor (for)
try: / except: sLaLemenLs
no swlLch sLaLemenL
lnsLead use of ifelifelifelse
See L3103 for ln-depLh dlscusslon
yLhon Language - Collecuons
SLrlng, llsLs, and Luples are -&.$&/*&-
LlsLs - uynamlc arrays (lndexlng, sub-ranges)
Slmllar Lo ascal arrays
MuLable collecuon of daLa of Lhe same Lype
1uples - llghL-welghL, lmmuLable llsLs
Slmllar Lo ascal records or C sLrucLs
lmmuLable collecuon of daLa of dlerenL Lype
ulcuonarles - Assoclauve arrays
keys musL be lmmuLable
yLhon Language - Collecuons
Collecuon lLeraLor for item in coll:
ulcuonary, llle and SLrlng are lLeraLors
Sllclng
nesung
Slngle-lLem Luple
yLhon Language - Advanced
CeneraLor funcuon - reLurns an lLeraLor
def generateItems(seq):
for item in seq:
yield 'item: %s' % item
LlsL comprehenslon
1here ls some more . (yLhon 101, books)
8unnlng yLhon code
lnLerpreLer vs. Compller dlscusslon
Source code ls complled Lo some klnd of byLe
code and execuLed by a vlrLual machlne
CyLhon: Source code -> yLhon byLe code
vM wrluen ln C, sLandard lmplemenLauon
!yLhon: Source code -> !ava byLe code
!vM as runume envlronmenL
lronyLhon: Source code -> lL (.nL1") byLe code
(.nL1") CL8 as runume envlronmenL
yy: 8unume envlronmenL wrluen ln yLhon
SLarung CyLhon
maln() 0"#$%&-1(234"/5* WlnMaln() 67189/0)9/5*
y_Maln() 0"#$%&-10)9/5*
y_lnluallze() 6234"/1(234"/:$/5*
ySys_SeLArgv() 6234"/1-2-!"#$%&5*
y8un_SlmpleSLrlngllags()
6234"/1(234"/:$/5*
8unModule() 0"#$%&-10)9/5*
y8un_AnyllleLxllags()
6234"/1(234"/:$/5*
y8un_lnLeracuveLoopllags()
y8un_lnLeracuveCnellags()
python -c
python -m
python f.py
python
y8un_SlmplellleLxllags()
.
y8un_llleLxllags()
.
yarser_AS1lromllle()
run_pyc_le()
CyLhon Compller & lnLerpreLer
def f():
a,b=b,a
prlnL ,Pello
Source code
(foo.py)
CALL_lunC1lCn
C_1C
8C1_1WC
8ln1_l1LM
...
Python byte code
Compller
lnLerpreLer
62;<=&*3> -3)*+
Abstract
Syntax Tree
>>> prlnL ,Pello
Interactive mode
734928174936
089362093843
692381298979
foo.pyc
CyLhon 2.3
arser Lransforms Lhe sLrlngs of Lokens lnLo a
concreLe synLax Lree (LCS1)
Lexer and LL(1) parser
1ransformer package Lhen LranslaLes Lhls lnLo a
more genulne AS1 (compiler.transformer)
lnpuL Lo byLe code compller ln compile.c
8yLe code ls generaLed by parslng Lhls AS1
vlslLor pauern allows hooklng ln opumlzauons
Compllauon resulL for lmporLed code
(noL Lop-level le) ls sLored ln .pyc le
yLhon AS1 (yLhon.asdl)
8epresenLauon of Lhe program wlLhouL code
As usual, every node ln Lhe AS1 represenLs a
synLacuc consLrucL
1ree rooL ls Lhe Module ob[ecL
SLaLemenLs (lf, Whlle, WlLh, rlnL, Class, Clobal,
8eLurn, 8alse, lmporL, Lxec, .)
Lxpresslons (8lnCp, ulcL, ?leld, SLr, .)
Speclallzed Lypes (excepLhandler, allas, sllce, .)
yLhon vM
vlrLual machlne parL of lnLerpreLer execuLes
yLhon byLe code
Slmple sLack machlne
PyObject* sLack - byLe codes operaLe on ob[ecLs
SLack frames are allocaLed on Lhe heap
C sLack frames polnL Lo heap sLack frames
Some hlgh-level byLe codes ("8ln1")
vM knows nearly noLhlng abouL C
represenLauon of speclc yLhon Lypes
yLhon ob[ecLs know noLhlng abouL Lhe vM
Cb[ecLs ln Lhe 8unume
Lvery yLhon ob[ecL lnsLance has a correspondlng
C Lype lnsLance
Lven baslc Lypes are allocaLed on Lhe heap
All C Lypes conLaln PyObject_HEAD sLrucL
members (e.g. see intobject.h / object.h)
8unume can Lherefore LreaL every yLhon ob[ecL as
varlable of Lhe Lype PyObject*
PyObject_HEAD conLalns number of references
on Lhe ob[ecL (ob->ob_refcnt)
C modules need Lo use Py_INCREF(obj) and
frlends
Cb[ecLs ln Lhe 8unume
ob_type member of PyObject_HEAD
holds polnLer Lo PyTypeObject sLrucL
ueLermlnes C funcuons Lo call on acuvlues wlLh
LhaL ob[ecL - 32(& !&34"#-
Some operauons are noL feaslble for some Lypes
PyTypeObject PyInt_Type = {
PyVarObject_HEAD_INIT(&PyType_Type, 0)
"int",
sizeof(PyIntObject),
0,
(destructor)int_dealloc, /* tp_dealloc */
(printfunc)int_print, /* tp_print */
0, /* tp_getattr */
0, /* tp_setattr */
(cmpfunc)int_compare, /* tp_compare */

lnLerpreLer Loop (ceval.c)
8lg swlLch sLaLemenL, worklng on an lncomlng
PyFrameObject* sLrucLure
Many opcode lmplemenLauons call C helper
funcuons (see PyEval_EvalFrameEx)
lor every byLe code lnsLrucuon, managemenL
of reference counung needed
lew byLe code predlcuon mechanlsm
L.g.: 7;06?@AB;6 oen followed by CD06BEFB>
Successful lnLerpreLer predlcuon mlghL lead Lo
successful processor branch predlcuon
lrame Cb[ecLs
PyFrameObject - argumenL for lnLerpreLer
ConLalns caller frame, code, varlables, and reLurn address
LxecuLed by Lhe lnLerpreLer loop
8epresenLs an anonymous *"#& "<=&*3 9/ &G&*$H"/
luncuon calls are mapped Lo frames
arameLers of funcuon calls are mapped Lo local
varlables ln a frame
luncuons are a programmlng language consLrucL,
lndependenL from frame concepL
luncuon ob[ecLs
luncuon denluon (def) blnds funcuon name
ln local namespace Lo funcuon ob[ecL
luncuon ob[ecLs are rsL class ob[ecLs
ConLalns defaulL value argumenLs
CompuLed aL denluon ume, whlch ls relevanL
when uslng muLable ob[ecLs
luncuons sLore Lhelr body as code ob[ecL ln
Lhe I$/*B*"#& aurlbuLe
Code ob[ecLs
lmmuLable ob[ecL represenung execuLable
byLe code
*"B/)!&: luncuon name
*"B/%"*)%-J *"BK):/)!&-: Local varlables
*"B/)!&-: names used ln Lhe byLe code
*"BI:&&K):-: names of free varlables
*"B*"#&J *"B*"/-3-J *"BL%&/)!&J *"BL:-3%9/&/"J M
ConLalns no reference Lo muLable ob[ecLs
Modules and lnLeracuve mode rely on
separauon of code and funcuons
yLhon 8yLe Code
uocumenLed ln DIS module
Speclal Lermlnology
N;O: 1op-of-sLack lLem
N;OP: Second Lop-mosL sLack lLem
N;OQ: 1hlrd Lop-mosL sLack lLem
Cperauons normally Lake Lop lLem(s) from sLack
and puL back Lhe resulL lLem
8aslc sLack operauons are lnllne C code
8aslc operauons on prlmluve Lypes are usually
Lhread-safe
8aslc SLack Cperauon
6;6BN;6: 8emove N;O
lmplles decreaslng of 1CS reference counLer
@;NBN8;J @;NBNR@AAJ @;NBF;D@
8oLaLe up Lhe Lop-mosL sLack lLems
e.g. Luple unpacklng : a,b=b,a
SD6BN;6: uupllcaLe 1CS reference
unary operauons on 1CS (e.g. DT?@UB6;OENEVA)
8lnary operauons on 1CS and 1CS1
(WET?@UB6;8A@J B0DXNE6XUJ BSEVESAJ MY
varlables
ulcuonary for varlables ln a scope
Lookup for "x" needs Lo check ln local scope,
global scope and bullL-ln scope
Local varlables sLaucally decldable by compller
8ule: All asslgned varlables are local
(asslgnmenL operaLor, lmporL)
Can be accessed by lndex, lnsLead of dlcL lookup
x = 1
def f():
y = x+1
varlables
ON;@AB1X;?SB1SAXANABZX;W?X /)!&
Access Lo global varlables
ON;@AB 1 X;?SB 1 SAXANABF?ON K):/$!
lasL access by uslng array of local varlables
uses *"BK):/)!&-[K):B/$!\ array from code ob[ecL
ON;@AB 1 X;?SB 1 SAXANABT?0A /)!&9
used when scope ls noL deLermlnable (e.g. 9!(":3 ln
funcuon)
Code ob[ecL has *"B/)!&- llsL wlLh name lndlces
Compllers uses *_lAS1 / *_CLC8AL whenever posslble
ON;@AB 1 X;?SB 1 SAXANAB?NN@ /)!9
AurlbuLes names are also sLored ln *"B/)!&- llsL
X;?SB 17;TON *"/-H
8elles on *"B*"/-3-[*"/-H\ llsL of consLanLs ln code ob[ecL
ln-lace / Sllces
ln-place operauons are llke blnary operauons,
buL demand a 'self-muLable' ob[ecL on 1CS1
ET6X?7AB6;8A@J ET6X?7AB0DXNE6XUJ M
1CS1 Lhen becomes new 1CS
lor operauons such as +=, -= (see L 203)
Sllce operauons
OXE7A]^: 1CS=1CS[:]
OXE7A]P: 1CS=1CS1[1CS:]
OXE7A]Q: 1CS=1CS1[:1CS]
OXE7A]_: 1CS=1CS2[1CS1:1CS]
Sllces / LlsLs
Sllce asslgnmenL
Changes 1CS / 1CS1 / 1CS2 ln-place
ON;@ABOXE7A]^: 1CS[:]=1CS1
ON;@ABOXE7A]P: 1CS1[1CS:]=1CS2
SAXANABOXE7A].
ON;@ABODWO7@: 1CS1[1CS]=1CS2
SAXANABODWO7@` del 1CS1[1CS]
Collecuon Lypes ln yLhon are cheap and fasL !
rlnung
6@ETNBAa6@ : lmplemenLs expresslon
sLaLemenL for lnLeracuve mode
6@ETNBENA0: rlnL 1CS Lo sys.stdout
6@ETNBENA0BN;: rlnL 1CS1 Lo 1CS le-llke ob[ecL
needs Lo conslder unlcode and so space
6@ETNBTA8XETAJ 6@ETNBTA8XETABN;
ConLrol SLrucLures
W@A?bBX;;6J 7;TNETDABX;;6`
Lnds Lhe currenL block
@AND@T BV?XDA`
8eLurn wlLh 1CS Lo Lhe caller of a funcuon
UEAXSBV?XDA`
8emove 1CS and yleld lL from a generaLor
ATSBFET?XXU` lnLerpreLer Lhen elLher re-ralses
excepuon, reLurns from funcuon or conunues
CD06BF;@8?@S #&%3)J CD06BEFBN@DA #&%3)J
CD06BEFBF?XOA #&%3)J CD06B?WO;XDNA 3):,&3
ulrecLly mapped concepLs
AaA7BON0N: run yLhon code
XEONB?66ATS` fasL llsL comprehenslon
E06;@NBON?@: from [module[1CS] lmporL *
DT6?7bBOAcDAT7A *"$/3: unpack 1CS
F;@BENA@ : 1reaL 1CS as lLeraLor, call /&G3dY
meLhod, push ylelded value on sLack
@?EOABV?@?@ZO: 8alses an excepuon,
parameLers are on sLack
ulrecLly mapped concepLs
WDEXSBND6XA /
CreaLe Luple of n Lop-mosL lLems and push resulL
WDEXSBXEON /
WDEXSB0?6
CreaLe empLy dlcuonary on Lhe sLack
WDEXSB7X?OO ` CreaLes a new class ob[ecL
1CS: ulcuonary of meLhods
1CS1: 1uple of base class names
1CS2: Class name
CALL_lunC1lCn argc
7?XXBFDT7NE;T ):,*
):,* low byLe conLalns number of posluonal
parameLers
):,* hlgh byLe conLalns number of keyword
parameLers
arameLers lLself are on Lhe sLack, aer Lhem Lhe
funcuon ob[ecL Lo call
0?bABFDT7NE;T ):,*
ushes new funcuon ob[ecL, 1CS ls Lhe code, ):,*
defaulL parameLers are below 1CS
And a loL more .. (read ceval.c)
lnLroduclng a new byLe code
LxLend omclal llsL of opcodes
(Include/opcode.h;
Lib/opcode.py)
lncrease MAClC number
(Python/import.c)
Change AS1->byLe code compller
(Python/compiler.c)
Change byLe code lnLerpreLer
(Python/ceval.c)
1hreadlng ln Lhe vM
yLhon Lhreads are Lrue CS Lhreads
Clobal lnLerpreLer lock
Cnly one (yLhon) Lhread ls runnlng lnLerpreLer
code aL Lhe same ume
8egular conLexL swlLch aer a number of execuLed
lnsLrucuons or wlLh long-runnlng operauons
roblem wlLh mulu-core Cu's
8eleased for long-runnlng C code (e.g. sysLem call)
erformance advanLage
8yLe code lnsLrucuons are aLomlc
SLackless yLhon
Lvery funcuon call creaLes a C sLack frame
Subrouunes vs. corouunes
yLhon funcuons can acL as corouunes or
3)-+%&3-, concurrenLly execuLed by scheduler
SupporL for channel communlcauon
AcL as llghLwelghL Lhreads
8edeslgned lnLerpreLer loop Lo avold Lhe C
sLack frame creauon on funcuon call
Memory ManagemenL
8AM Swap
Cperaung SysLem vMM (page based)
CS memory allocaLor ( malloc() )
Peap allocauon wrapper (pymem.h)
PyMem_Alloc,
PyMem_Free, .
Cb[ecL AllocaLor (obmalloc.c)
PyObject_Malloc,
PyObject_Free,
Cb[ecL-speclc allocaLors
(e.g. lnLob[ecL.c)
C code of lnLerpreLer / modules
Cenerlc ob[ecL managemenL (ob[ecL.c)
PyObject_New, PyObject_Del,
Cb[ecL AllocaLor
8y deslgn, many very small allocauon requesLs
LveryLhlng ls an ob[ecL !
Speclal opumlzauon for performance
(obmalloc.c)
8equesLs >236 byLes handled by malloc
Smaller requesLs slzes are grouped (8 byLes aparL)
Memory pools of 4k lengLh each (vMM page slze),
wlLh own free llsL
ools are used by dlerenL requesL slze allocaLors
8 8yLe allgnmenL of reLurned address
Carbage CollecLor
1radluonal garbage collecuon (e.g. mark and
sweep) would demand a seL of rooL ob[ecLs
LxLenslon modules can creaLe own yLhon ob[ecLs
CC for allocaLed C ob[ecLs noL really porLable
1raverslng all ob[ecLs ls expenslve
lnsLead: Slmple reference counung
ln ob->ob_refcnt from PyObject_HEAD
Works wlLh every malloc() / free()
Py_DECREF() - (ob[ecL.h)
Calls nallzer when reference counL comes Lo zero
Carbage CollecLor
luncuons LhaL creaLe an ob[ecL seL Lhe
ob_refcnt Lo 1, and sLore lL - or desLroy lL
by calllng Py_DECREF
Some sLore funcuons Lherefore don'L lncrease Lhe
reference counLer (e.g. PyList_SetItem())
Cb[ecLs can be sLucked ln Lracebacks
Weakref module (L 203)
lor ob[ecL caches (weak dlcluonarles)
lor clrcular references (uCM node relauons)
Clrcular 8eferences
Pead
LlsL elemenL
ob_refcnL=2
LlsL elemenL
ob_refcnL=1
LlsL elemenL
ob_refcnL=1
Pead
LlsL elemenL
ob_refcnL=1
LlsL elemenL
ob_refcnL=1
LlsL elemenL
ob_refcnL=1
Cycllc Carbage CollecLor
8eference cycle: unused ob[ecL(s) even
Lhough reference counLer ls noL zero
1esL ls only relevanL for conLalner Lypes
usage of double-llnked llsL of all conLalner
ob[ecLs (gc_next , gc_pref)
ueLermlne all conLalners whlch are only
referenced by Lhem self
Cb[ecLs ln cycles wlLh nallzers __del()__
are added Lo seL of uncollecLable ob[ecLs
Crder of nallzer calls ln Lhe cycle unclear
Sources
use Lhe source, Luke (yLhon Svn Lrunk, March 2008)
Mark LuLz, rogrammlng yLhon. C'8ellly 2006
hup://mall.pyLhon.org/plpermall/pyLhon-llsL
hup://www.pyLhon.org - L's, yLhon 1uLorlal,
LxLendlng and Lmbeddlng Lhe yLhon lnLerpreLer 1uLorlal, yLhon lAC
hup://docs.pyLhon.org/llb/byLecodes.hLml
hup://docs.pyLhon.org/apl/Lhreads.hLml
hup://www.voldspace.org.uk/pyLhon/arucles/code_blocks.shLml
hup://eMoL.org/pyref/Lype-code.hLm
hup://www.devshed.com/c/a/yLhon/Pow-yLhon-8uns-rograms/
hup://www.rexx.com/dkuhlman/pyLhon_101/pyLhon_101.hLml
hup://arcLrlx.com/nas/pyLhon/gc/
hup://www.sLanford.edu/class/cs242/slldes/2006/pyLhon-van8ossum.pdf