oe ID: 6689691 ()G)-Pabe 8 Instructor Notes nd about 2 years ago by| (U/FOUO) Instructor notes for COMP 3321 UNCLASSIFIEDHFOR- GFFIGIAL BSE-ONEY (U) So, youre teaching the Python class. What have you gotten yourself into? You should probably take a few moments (or possibly a few days) to reconsider the life choices that have put you in this position, (U) Course Structure (U) As mentioned inthe involution, this course is designed for fexcbilty. When aught in a classroom setting, single lesson or module can be ‘covered ina session that lasts between 45 and 90 minutes, depending on the topics to be covered. The standard way to structure the course is as fulltime, two week block. During the frst week, the ten lessons are covered with moming and aftemioon lectures. During the second week, up to ‘ten modules are covered ina similar manner, as needed or requested by the students in the class. (the class needs are not known, take @ vote), During the fist few days of class, students should choose a project to work on. On the lat day, students should report back onthetr progress and, if possible, demonstrate ther work. Instructors should be available outside of lactures to assist students with exercises and projects. {(U) The two week block isnot the only way of teaching the course. The material could be presented at a more leisurely pace, for instance during a ‘weekly brown bag lunch that continues for several months. Alternatively, f students are already prepared (or wiling to do some ofthe intial lessons. ina selstudy manner), a great deal can be accomplished in a two oF three day workshop. For instance, (all students already have a basic ‘knowledge of Python, they might well start with the lessons on iooling and wiling.masiules anc packages, then move on to cover various modules of interest. Approved for Ri 36oc ID: 6689691 (U) Instructional Style (U) When teaching mathematics, the common practice ofthe instructor wring solutions on the chalkboard is a moderating method that helps students keep up. Writing on a chalkboard isnot usualy helpful when teaching programming, but the same principle applies; as the instructor, you should adopt practices that help you slow down. We recommend that you have alive, interactive session displayed a the front ofthe room, large ‘enough fo ll the students to soe. This session can either be a terminal session ora Jupyter notebook. The important detallis that in most cases the commands should not be pre-populated; eg. you should not just execute cll from an existing Jupyler notebook. The materials are present to help you prepare, and as a reference for the students as they work on exercises; they are not an acceptable substtue forthe shared experience of teaching and learning. You will make unexpected mistakes as you write code lve In font ofthe lass. Dont worry, relax, and let the students help ‘you-it wil help them eam the principles and figure out how to solve their own problems, This Is not a substitute for proper preparation; too ‘many mistakes and fumbles will cause your students o lone interest nthe course and trust in you (U) Ongoing Development ae (U#FOUE} The developers of this course believe thatthe current materials are sufficiently welldevetpied tobe an effective ad forthe course. However, improvements, extensions, and refinements are aways welcome, To that ert we have attempted to make i easy to contribute to the Drie. Th docmartan bao on te Mack offal COMP 2921 mete, you weno neko charge for our ‘own purposes, fel tree to clone that repository or fork any ‘on the Juptyer Gallery. Please submit @ change request onthe Gallery if you'd like to make a one-off contribution, including new or improved exercises, additions to lessons or modules, or entrely new lessons or ‘modules. Ifyou woul ke to be a collaborator onal ofthe COMP 3321 notebooks, contact the COMP3321 GlobalMe group and ask o be added. (U) A possible icebreaker (U) To got the students interacting with each other as wel as thinking about code at an abstract level, consider the following icebreaker. Instructor becomes human compile to interpret written instructions to get out ofthe room. (U) Phase I~ discussion with whole class (U) Invent a programming language together, one sufcient fr ths task. Clearly explain the task, showing where inthe room the instructor wil begin and how big a steps, (U) Take suggestions from students about what instructions they will want to use. Write each instruction on the board. Be clear that everything ‘written on the paper must come from syntax onthe board, No other syntax allowed, (U) Make sure the instructor and students agree about precisely what each instruction means. AS the instructor, be certain thelist onthe board is. ‘enough for you to solve the problem. Give hints unt i's complete (U) Let the students come up with the syntax. But here are some syntax examples they may come up with + step(n) takes in an integer and causes instructor to taken steps,Doe 6589691 um(d) ~ takes in a number in degrees and caused instructor to tum clockwise that many degrees. Students may abuse this and putin numbers that lead to dizzyness, e.g. tun(1440) + obstacle ~ returns boolean indicating rather an obstacle is directly in font of instructor. Variations for checking tothe left or right may be Sesirable as wel, + If-<> then < > else < > ~ fst blank takes boolean (make sure boolean functions exist), Second blanks take instructions, + while <>: <> ~ takes boolean function and any expression not ~ expression to reverse a boolean any integer (U) Phase It break into teams (U) Teams of 35 students tend to be appropriat (U) Each team produces a piece of paper wih computer instructions forthe human compiler to get out ofthe room. Only allowable syntaxis what is writen on the board. Should take no more than 15 minutes. (U) Phase I~ demonstrations (U) One ata time, the instructor takes a team's solution and follows the instructions iterally and fairy. Does the instructor get out ofthe room? (U) Introductory e-mail (U) Tere area couple things it woud be nice ifthe students could nave done in advance, n pacar having GITLAB accounts and INHERE Agreements. | have the COMP3321 learning facitatorssond te folowing eal to enrolled students (U) Aone (U) You are receiving this e-mail because you are registered for COMP3321 beginning <.>. (U) fatal possible, we could use you do a few setup things in advance to make class go smoothly onthe frst day. Really, two simple things that ill take @ minute of your time and save us hours on the fst day of class. (U) 1) GO iagree and find, read and agree to the INHERE user agreement (U)2)60 gitab. (U) in more deta: (U) 1) The course wil be run-vtaLABBENCH and NBGALLERY. Before using this, you will need to agree,to'the terms of service and| knowledge ths Instuctons can be found ber: noallarsa acid (GO NBGALLERY) by cleking GET A BENCH. We ony TOTS foto theft, uta below. Be careful going fre because LABBENCH modi ea dest wo weeks ar creation 50 Yeu want aes one the fret ay of lags : (U) GO iagree. Search "inhere”. Read the INHERE user agreement and agree toi co eventually acknowledge this agreement. That's what we needDoe ID: 6 {85.0 ota. 2y ana tee one an acount il be eel. Thats al we need bet (Utyouwantto go turer and ue gta he command ine[_——Jtownstctons har snide (U) Tre are several opto presented. Whatever you carga won Ws, (U)3) Optpna. Th lao bt 3'CABBENCH. But some people retro use[_——JoachiThs wl ake goths wrk. nial frwconda oy fekowng bese isbn: ioscan rade 8 Nan 22673 (U) you ave any issues wih he instuctons, please contact he insigor: > a (U) Submitting projects - (Aone mr api onc css [Trt an oer Gagan costo Poney epee mand ft BST SET nm VY. Stes wl aut ace © Ges. an bo ean aSumplan fresh he web epleaton by nee" rand copying and esi ere. The web option owt outa tar Steno te Suse ta aed bt can ee Bun comune a rye UNCLASSIFIEDHFOR-OFFIGIAL USE-ONLYoc ID: 6489692 Python Programming (U#2YO} Course introduction and syllabus for COMP 3321, Python Programming. UNCLASSIFIED//FOR-OFFIGIAL USE-ONLY— (U) History .-in December, 1989, | was looking for a "hobby" programming project that would keep me occupied during the week around Christmas. My office ... would be closed, but | had a home computer, and not much else on my hands. | decided to write an interpreter for the new scripting language | had been thinking about lately: a descendent of ABC that would appeal to Unix/C hackers. | chose Python as a working ttle for the project, being ina slightly irreverent mood (and a big fan of Monty Python's Flying Circus) Guido van Rossum, Foreword for Programming Python, 1st Edition. (U) Motivation (U) Python was designed to be easy and inttive without sacrificing power, open source, and suitable for everyday tasks, with quick development times. I makes the layers between programming and problem solvingseem as thin as possible. Its suitable for- ‘+ Opening an interactive session to solve the Dally Piz, + Writing a script that automates a tedious and time-ceinhithDo ID: 6689492, Creating a quick web service or an extensive web application, and + Doing advanced matnematica esearch. (U) Ifyou dont know any programming languages yet, Python isa good place to start. It you already know a diferent language, its easy to pick Python up on the side. Python isnt entirely free o frustration and confusion, but hopefully you can avoid those pars until ong after you get some ‘900d use out of Python, (U) Programming is nota spectator sport! The more you practice programming, the more you will eam in this class, both in breadth and depth. Python practically teaches itsetf-the goal of your instructor is fo guide you to the good parts and help you move just ait bt more quickly than you would otherwise. Happy Programming! Betas reen (U) Objective (U) The goal ofthis cass is to help students accomplish work tasks more easly and robustly by programming in Python. To pass the course, each ‘student must wrte atleast one Python program that has substantial personal ult ors of significant personal interest. When choosing a project, ‘students are encouraged to fit think of work-related tasks. For students who need help geting stared, several suggestions for possible projects ‘are found atthe bottom ofthis page. On the fist day, instructors wil lead a discussion where project ideas are discussed, {(U) This class is designed for students of varying backgrounds and levels of experience, from complete navi to competent programmer. Asking ‘each student to design and implement their own project allows everyone to leam and progress at an individual pace. (U) Logistics (U#F946) This course is designed to be suitable for set1eaming. Even ifno formal offerings are available for your schedule, you may access and work on the modules of his course at anytime. Even if you dont have access toa recent version of Python on a workstation or vital machine, you ‘can access a personalized Jupyter notebook avaliable on | ABBENCH. Foran individual pursuing this self-study option itis recommended to fist ‘over the Python Basics roughiy in order, then select as many ofthe Useful Modules as seem appropriate. You can also use this Jupyter ‘notebook to experiment and wrie solutons to exercises. (U##O4E) One possibilty fora group of potential students who start out with a diferent amounts of programming experience is to use Jupyter as a sel-study tool until everyone has a basic understanding of programming, then folow up with an abbreviated instructor-led course (anywhere fom two days toa week or more, depending on needs). (U) In the Classroom (U) For a two week course: there wll be a morning lecture and an afternoon lecture every day. The moming lecture wil ast between an hour and hinety minutes; the aftemoon lecture wil be somewhat shorter. Ha lecture Is going too fast, please ask questions to slow us down! Its going ‘00 stow feo! fee to work ahead on your own, Please Enter Footer Text.oc ID: 6499692 (Gi#SL0} You wil ether use Python within jronment or within LABBENCH, SSS ews] ‘ut some diferences between the Python 3x and 2x les, Tis Course wil focus on Python ‘Zryou need oF wantTo rn Pyihon oh Linux, probably withing achineShon VM, well work wih you. To the extent possible, we wil write code in platform-agnostc manner and pont out fetes that are unique specif versions of Python. : {(U)Atnough lectures wil ony take up two o three FoUs.each day, encourage you to spend the remainder of yur day programming in Python, either on your own orn groups, butin the classroom f possible. AL least ofe instructor willbe available in the classfoom during normal business hours. i (U) This course is a work in progress; we welcome all suggestions. There afe'mgre Ueful Modules than we can Kispe to cover ina two-week class; instructors wil ake a vote to determine which modules to cover. I there is anbihe topic that you would lke {9 have covered, especialy along the lines of "How would | do x, v oF 2 in Python?" please ask-if there's enough interest; wel covert ina lecture. Weld even be happy to have you contribute tothe course documentation! Tak to an instructor to find out how. (U) Table of Contents (U) Part |: Python Basics (Week 1) + (U) Lesson 01: Introduction: Your First Python Program + (U) Lesson 02: Variables and Functions © (U) Optional: Vasable Exercises © (U) Optional: Function Exercises + (U) Lesson 03: Flow Control © (U) Optional Fw Control Exercises (U) Lesson 04: Container Data Tynes ‘| + (U) Lesson 08: Modules. Namespaces and Packages © (U) Supplement: Modules and Packaoes + (U) Lesson 08: Exceptions, Profling and Testing + (U) Lesson 10: teraors. Generators and Duck Typing © (U) Supplement: Pioeining wih Generalors + (U) Lesson 11: Sting Formatting=“) Part I: Useful Modules (Week 2) + U) EERE Module: Cotectons and tertoois @ (U) Supplement: Functional Programming + (U) Supplement: Recursion Examales una ‘© (U) Supplement: Maninulating Microsoft Office Documents wih win’2com + (U) Module: Threading and Subprocesses + (U#FOweYDstdbutig a Python Package at NSA + (U) Modula: Machine Learning ntoduction (U) Homework + (U)Day1 Homework Please Enter Footer Text...‘Doe ID: 6689692, * {Uy Day.2Homewors (U) Exercises (with Solutions) (U) Class Projects (U) Glick nore to get to @ notebook containing instructions for password checker and password generator projects. (U) Project Ideas + (U) White a currency conversion script + (U) Write a web application + (U)DoProject Euler problems + (U)Eind Anomalous Acivily onthe BigCorm Network + (U)BSA Encrymtion Module, Par 2 + (U) Pick project rom one of the Safari books below (U) General Resources (U) Python Language Documentation (UHFEYE}NSA Course Materials(©) Dive into Python (U) Exe ython Programming (Head First Python (U)HHion Performance Pytnon (advanced) ()Leaming Python {(U) Leaming Python Programming (videos) + (U)Prooramming Python + (U)2ytnon Crash Course (includes 3 sample projects) + (U)2ytion Piayoround (more sample projects) (U) 2ython Packet Reference (consider geting a print copy) (U) 2ython Programming fr the Absolute Beginner (even more sample projects, games & quizzes) * (U) and More Python Programming forthe Absolute Beginner + (U)Thing Python + (U) Safar Books (General Query) (U) Other + (U)Einal Project Schedule Generator + (U) Justa ite notebook fr randomly generating a schedule for students to present their nal projects for COMPS321, (U#FOUE) The Pyinon coun on NSA Gitkat (U) The Hitchhiker's Guide to Python! (U#FOVO) Pyinon on Wisinio (U) Python on StackOveriow (U) Addtional targeted resources (often excerps from the'sbove)aredinkédn eactriesson and module. ere 86Doe UNCLASSIFIEDHFOR OFFICIAL USE-ONLY-oe ID; 6689693 Lesson 01: Introduction: Your First Python program a (U) Covers Anaconda installation, the python interpreter, basic data types, running code, and some built-ins. UNCLASSIFIED//FOR-GFRHIGIALUSE-ONEY (U) Welcome To Class! (U) Lets get to know each other. Stand up and wai for instruction, (U) Who has a specie project in mind? (U) Method 1: Anaconda Setup (U) Aternately, flow tis tradecrat hub arte, ios production \radecrafproj.nsa ie. govlentry/29475 (U) We will be using version 4.4.0 ofthe Anaconda Python dstrbution, availabe from Git Software. Anaconda includes many packages for large~ ‘scale data processing, predictive analytics, and scientific computing (U) Installation Instructions ‘Approved for Release by NSA on 12-02-2019, FOIA Case # 108165es EU) Click onthe lnk above 2.{U) Glick on the “Download Now” button 53. (U) Accep the agreement and click "Next 4.(U) Click Next 5. (U) Download "Anaconda3-4.8.0-Windows-x86_64 exe 6-(U) Open the folder containing the download (pica ths is your “Downloads” folder) 7 (U) Doublecick the Anacondas-4.4.0-Windows-186_64.exe le 8 (U) Click "Nex?" to Start the installer 8. (U) Glick the" Agree” button to accept the icense agreement 10.(U) Choose to install for “Just Me" and cick "Next™ 11, (UHFOUS) Select a destination folder on your U:crive (such as U\prvatelanacondad) 4. (U#EOUO} Cick the "Browse." butlon, Gck on "Computer" and selec the U: drive 2 (UAFOUO} Select "My Documents" and press "OK" [Note: DO NOT Make a folder named anaconda3) '3. (UAFOUE} n the Destination Folder input area add “anacondad" asthe folder name 4. (UMPUE}Clck Next 12.(U) Cick instal to begin the install eave checkboxes a fs) 13 (U) Wait about 30 minutes for the instal complete 5 (U) Running python deceeeeeee? (U) You can run python directly on yourfET Pisskop and immediately interact with it 4. (U) Open a Windows command window (Type “cma” in the Windows Programs search bar) 2 (U) Type “python” inthe command window (U) Running Jupyter (U) Aterately, you can run python in a browser from a web-enabled python called a jupyter notebook, Tit Wc Gory: "or Wis Cass, however, we each ‘Sar up aur own ndiiGual Jupyler web: portal ofan Our Glass novebooks. 4. (U) From the Windows Start menu, search for "upyter 2 (U) Right-click on Jupyter Notebook inthe resulis and select Properties 3. (UHFOUS) in the “Target field, add" uprivat” atthe end (ater "notebook’) [Note: don't forget the space before u:prvate] 4 (U) Chick Apply and then OK 5. (U) Search for jupyter again in the star menu and click on Jupyter Notebook to run it 6. (U) Wait a few moments...This should launch Jupyter in your browser at ili. localhost 8888 (U) Method 2: LABBENCH Setupoc ID: 6589693 (UHFEUSE} Step 1: Access to LABBENCH + (U#FOUO) 90 inoree and read and accept the INHERE User Agreement. (U#FEUO} This s a prerequisite for access to LABBENCH, a VM system where we willbe working. It may take a few hours forthe approval to propagate through the system. (U) Step 2: Visit Jupyter Gallery 41. (UAFOUO} 90 junviee 2 (U) Glick on the Jupyter Gallory logo to get to the Gallery 3. (U) Click on Tour the Gallery for quick demo. 4. (UFOS) To find the course notebooks, either search for “Syllabus” or choose Notebooks > Learning > COMP 3321 and sor by title to find the Syllabus. (U#FOUG} Step 3: Set up Jupyter on LABBENCH (U#FOUO) At the Jupyter Gallery, click “Jupyter on LABBENCH™ for a tutorial on howto get set up. (U) Basic Basics: Data and Operations (U) The most basic data types in Python are: + Numbers © Integer (these are “arbitrary precision"; no need to worry whether i's 92 bits or 64 bits, etc) © Float (using 1) forthe imaginary number) + Stings © No iflerence between single and double quotes © Escape special characters (e.g. quotation marks) © Raw sting r'raw string” prevents need for some escapes, © Thiple-quotes allow mutiple line strings © Unicode u'aert \x26 Ernie’ ctype ‘unicode'> + Booleans: true and False (U) We operate on data using + operators, e.g. mathematical operators +, - ; the Keyword in, and othersDeclD: 62 functions, which are operations that ake one or more pieces of data as arguments, ©, ‘ype("heLlo") , Ien("worls") , and + methods, which are atached toa piece of data and called from itusing a. to separate the data from the method, e.g. "Hello World”. sp1ie() , oF *abe’ -upper() (U) Deep in the guts of Python, these are all essentially the same thing, but syntactically and pedagogically it makes sense to separate them, (U) Pisces of basic data can be stored inside containers, including + Late + Dictionaries + Sets ‘but well ntroduce those later. (U) The Interactive Interpreter \with that basic background, let's try some things in your Windows command window. Us\privaterpython Python 3.5.1 [Anacondo 2.5.0 (68-b{t)| (default, Jan 29 2016, 15:01:46) [KSC v.1900 64 bit (ANDSA)] on win32 Type “help”, “copyright”, “credits” or “license” for more inforsation. ‘typet5e7) sype(7-1 - 2.2) as 4 this changed tn pythons aB//s nelle” + * wort nelle * + 20 (U) Executing code in a fileeo I: 699603 Open the fle Fiest-progran. (U) Ifyou dont have a favorite editor do this: 41. (U) Go to your Jupytr porta at hit. /localnost 8888!roe 2 (U) Pull down the “New” button menu and choose “Text Fle" 3. (U) Click on the "Untitled! et" name and enter the new fle name as “ist-program py" (or anything ending with .y )n your favorite editor (Iuse enacs , but you can use whatever you want), {(U) Type some Python statements int so {(U) Don't forget to save it (File>Save from Jupyte). (U) Torun it, give the fle name as an argument to Python: ‘Make sure the command window is referencing the same folder as the fle. That is U\prvate for most. Hf your command window isnot referencing U:\pivate, do tis: 1 Enter "U 2. Enter “cd private* U:\privatespython Fist-progran.py (U) Nothing appears to happen, because auto-prnting ofthe output ofa function only happens in the interpreter. Fix it up: print(5e7) print(9r43) printcarsi2) (U) Built-in functions and methods (U) Some functions work on almost any arguments that you supply: ‘+ elp(x): shows interactive help ‘+ dir(x): gives the directory of the object, Le. al the methods available + typa(x) tls you the type of x —a type's almost the same as any other object + Ssinstance(a,0) tel fobject a is an instance of & , which must be a type ; something like type(s) = + print ‘+ hasater(a,b): tells whether a has something by the name ; something like & in din(a) + getatroc ID: 6559693, sa + snput (U) Constructor functions usually tr to do their best with the argments you give, and retum the appropriate data ofthe requested type: ‘+ str tums numbers (and other things) into their sting representations ‘+ Ant: tuncates float , parses str ings containing a single integer, with optional radix (Le. base), eror on complex + lost: parses strings, qlves oat representation of int , eroF on complex ‘+ complex: takes (real, imag) numeric arguments, or parses a str fora single number (U) Other functions only work with ane or two types of data + Numbers: ‘© Functions: abs, round, float , max, min, pow (modula), chr dimes, ete © Operators: Standard math, bitwise: <<, >>, ‘> Methods: Numeric classes don't have methods + Stings: © Functions: ten, ain, max, ord © Operators: +, * (with a number), 9 © Methods: strip, split, startswith, upper. find (U) Exercises: 1. Make a shopping list of ive things you need at the grocery store. Put each item on its own line in a cel. Remember to use quotes! Use print() so that each of your tems displays (ty it frst without). 2. Your groceries ring up as 9.42, 5 67, 3.25, 13.40, and 7.50 respectively. Use python as a handy calculator to add up these amounts 3. But wait You decide you to buy five of th last tom, Re-calculato your total, 4 Using the 1en() function, determine the number of characters inthe string "blood-oxygenation level dependent functional magnetic resonance imaging" (Fun fact: this string isthe longest entry in WordNet 1 index). 5. Pick your favorite snack. Use the * operator to print 100 copies of i. Modify your code to have them print with spaces between them. 6. Challenge: Run >> IsDivisibleBy7(21) Tne >>> isDivisibleBy7(25) False (U) White a function sspivisibiesy(num, divisor) to check if num is evenly divisible by divisor. >>> eDiveibleBy(35,7) True >>> isDivsibleBy(35,4) False (U)Make a function snoue(wora) that accepts a sting and returns tha >>> shout("bananas") ‘BANANAS? ting in capital letters with an exclamation mark. (U) Make a function introduce) to ask the user forthe name and shout it back to them. Call your function shout to make this happen. >>> What's your name? >>> Bob HIBo8!Dec ID: 6589693 Lesson 2 - Variable Exercises (U) Variable Exercises for COMP3321 Lesson 2 (U) Lesson 2 - Variables Exercises (U) Identity the type of each ofthe folowing variables, and add the type after each variable in a comment. > (U) What vale is in variable my_var at the end ofthese assignments? [Add a comparison after the last statement in the form of my_val =oc ID: 6689683, myvar = 99 yvar saa ry_var = ste(ay_var) myvar “= 2 sy_var = Len(ey_var) myvar *= 6oc ID: 6589693 Lesson Mase Flow Control peer gy voor e00 orf Ti Ge saat = (U) Python flow control wth conditionals and loops (i, wile, for, range, et). UNCLASSIFIED (U) Introduction (U) you have ever programmed before, you know one of the core building blocks of algorithms is flow contro. It tells your program what to do next based on the stat tis currently in (U) Comparisons (U) First. let's lok at how to compare values. The comparison operators are >, >=, <, <=, I=,and == ..When working with numbers, they do ‘what you think: tum True oF False depending on whether the statement is true oF fale.Doe ID: 6689493 (G)Byinon 2x wilt you try to compare any two objects, no matter how ifferent. The results may not be what you expect. Python 3.x only ‘compares types where a comparison operation has been defined. ‘apple’ > “orange” # cose-sensitive alphabetical ‘apple’ > ‘Orange’ ‘apple’ > (‘orange’] ‘apple’ > (‘orange’) (U) We wi leave more discussion of comparisons for later, including how to inteligently compare objects that you create (U) Exercises 1. Write @ you son function that randomly picks a number from your price ist (9.42, 5.67, 3.25, 12.40, and 7.50) and prints. Tese or False depending on whether the random number is greater than 10. 2 White a function snack check that takes string snack and retums ‘True Or False depending on whether ornotitis your favorite snack (U) Conditional Execution: The if Statement {(U) The i statement is an important and useful tol. basically says, "Ifa condition Is rue, do the requested operations" ‘def even(n): AF (982 == 0): ‘rint(*T an even!*) even(2) even(s) ‘even( hello") # tat wos silly (U) What if we want to be abe to say we are not even? Or the user submitted a bad type? We use else and elif clausesec ID: 6689693 ef even(n): AF (typen) I= Ant) print("T only talk about integers") alte (0 2 == 0): print(*T an even!") se: print(*T an oat") ceven(2) ceven(3) ceven(‘hel1o") (U) Exercises 1. Re-wnite the snack check to take a sting snack and prints an apporpriate response depending on whether the input is your favorite snack or not 2 Wiite an in_grocery_1ist function that takes in a grocery_sten prints a diferent message depending on whether grocery_iten isin your grocery list. 3. Modify in_grocery. list to testi grocery_tten isa sting. Print a message warring the user fits not 4 Challange: Re-write the you won function fo randomly choose a number from your pies list and print appropriale message depending on whether you won (the number was greater than 10) oF not. Also include the amount of change you willbe receiving in your message. (Recall you are winning the amount change you would have owed... 5. Advanced challonge: Write a function that imports datetine and uses it to determine the current time. This function should print appropriate message based onthe time exif the curent tie is between 0900 and 1000, print the message "Morning Lecture time! (U) Looping Behavior (U) The while Loop (U) The while is used for repeated operations that continue as long as an expression is tue. (U) The famous infinite loop: while (2 +2 == 4): prine(“forever") (U) Armistake that may ead to an infinite lop:ac ID: 6649693 ino wtde (1 <= 20): print(i) (U) The below is probably a more sensible thing to type. 1-0 wotte (1 rine 1) ed 2): (U) break and continue (U) For more contro, we can use break and continue (they work just a in C). The break command will break out ofthe ‘smallest while oF for loop: tee wetle(trve): ded int (i) reel ‘break 22) (U) The continue command wid halt the current iteration ofthe loop and continue tothe next value iso wide(trve): tea Af (== 10): eint("T an 101") continue print (i) AF (1 => 20): ‘breakoe ID: 6559693 (U) The else clause (U) You can also have an else statement atthe end ofa loop. It will be run only if the loop completes normally, that is, when the conditional ‘expression results in False .A break wil skip it. i-0 nite (1 <2): print) ted aise: print("This executes after the condition becones false.") print("Oone!") wntte (4 <2): print) Sf True: ‘break tea ease: Print("This won't print because the loop was exited early.) prine¢"Done!") (U) Exercises Hint you will not need continue oF break for these exercises. 1. Previously we printed out many copies ofa string using the * operator, Use a whe Toop to printout 10 copies of your favorite snack. Each ‘copy can be on it's own line, tha’ fe. 2 Mocand match! Write @ white loop that uses the to print multiple copies of your favorite snack per line. Print out 10 lines with the number ‘of copes per line corresponding tothe line number (your fist line wil have one copy and your last ne will have 10). 3. Challenge: Write a white loop that prints 100 copies of your favorite snack on one single (wrapped) ine. Hint use = (U) The for loop (U) The for loops probably the most used contol low element asi has the most functionality. It basically says, “forthe following expict items, {do something,” We are going to use the listtype here, More interesting properties of tis type wil flow in another lesson.oe ID: 6589688, for 4 im [2,2,344,55'3" print 1) (U) The variable 4 "becomes" each value ofthe ist and then the folowing code is executed for An (2,243,455, °8°,"b°,"e") print(s, type(s) for € Sn “ovat print c) (U) Exercises 1. Wite a for loop that pnts out each character inthe sting "blood oxygenation lave dependent functonal magnetic resonance imaging” {Fun fact ts ting i the lengest ety in WordNet Index). 2 Te your erocor a of ve ems (or cote ene), Writ opt cto the message "Nott sl, buy.“ and then tho grocery em. 3 Write @ for loop that prints outa numbered Ist of your grocery items, 4. Clearly your favorite snack is more important than the other items on your list. Modify your for loop from Exercise 310 use break sto printing once you have found your favorite snack in your list. Question: Could you have achieved the same result without using a break ? Bonus: if your snack isnt inthe lst, have your code print a warning at the end. 5 Challenge: use the string method split to write a. for loop tha pints aut each word inthe string "blood-oxygenation level dependent functional magnetic resonance imaging” Hint: run help(str- 18's just on object! for 4 in range( 10020): (Ln 2 = 0): print 1) b= range(2, 1090000, 100) » be) on be otal (U) ExercisesoF se cane to write a for loop to printout a numbered grocery list. 2 Use enumerate to print outa numbered grocery Ist. Youve now done this three ways. What are some pros and cons to each technique’? ‘There are often several different ways to get the same output! However, usually one is more elegant than the other. 3.Use range 10 wile @ for loop that prints out 10 copies of your favorite snack. How does this compare to useing a while loop? ‘4. Challenge: White a "Guess my number” game that generates a random number and gives your user a fixed number of guesses Use input to get the user's guesses. Think about wat loop type you might use and how you might provide feedback based on the user's uesses. Hint: what type does. input return? You might need fo conver this to a more useful type... However, now what happens if your User inputs something that srt a number? UNCLASSIFIEDoe ID: 6689653 Lesson 3 - Flow Control Exercises (U) Flow Control Exercises for COMP3321 Lesson 3 (U) Lesson 3 - Flow Control Exercises (U) Change the loop below so that i prints numbers from 1 to 10. for & An range(s) print (1) (U) Using a for loop and enumerate, write a function getindex(string, character) to recreate the string method .index “skyscraper” index(°c") etindex( “skyscraper, "c") ) Using the shout function from the fist set of basic exercises, wite a shout_words(sentence) function that takes a sting argument and "shouts* f2ach word on its own ne. shout_words("Everybody 1ikes bananas”) # eveRve00"! sans!Do ID: 659603 (G)Whte an exeract_longer(ength, sentence function that takes a sentence and word length, then retums alist ofthe sentence's words that ‘exceed the given length. I no words match the length, return False. extract_longer(S, “Try not to interrupt the speaker.”) # ["interrupt', 'Speoker."] lextract_longer(7, “Sorry about the mess.*) falseDoe ID: 6589693 Lesson 04: Container Data Types a] ss . | oo (U) Lesson 04: Container Data Types UNCLASSIFIED (U) Introduction (U) Now that we've worked with stings and numbers, we tum out attention to the next logical thing: data containers that allow us to buld up ‘complicated structures. There are diferent ways of puting data into containers, depending on what we need to do with, and Python has several builtin containers to support the most common use cases. Python's builtin container types include: 4. ust 2 wole 3 atet 4 set 5. frozenset (U)Ofthese, tuple and frozenset are immutable, which means that they can not be changed aftr they are created, whether that's by addition, removal, or Some other means. Numbers and stings are also immutable, which should make the following statement more sensible: the variable that names an immutable object can be reassigned, but the immutable object itself cart be changed. (U) To create an instance of any container, we cal ts name as a function (sometimes known as a constructor. With no arguments, we get an empty Instance, which isnt very useful for immutable types. Shortcuts for creating non-emply List 6, tuple 6, dict s, and even set s willbe covered in the folowing sections. ust()oo ID: 6689693 act) tuple() set) (U) Many bulvin functions and even some operators work with container types, where it makes sense. Later on weil see the behind:-the-scenes. "mechanism that makes this work; for now, well enumerate how this works as pat of the discussion of each separate type. (U) Lists (U)A ist is an ordered sequence of zero or more objects, which are often of cifferent types. I is commonly created by putting square brackets. [ } around a comma-separated isto is iia values: ‘spam’, “eggs, 5, 3.2, [100, 200, 300]] fruit = ['Apple’, ‘Orange’, "Pear, ‘Line (U) Values can be added to or removed from the lst in efferent ways: frult.append( Banana") frutt-insert(3, “eherry") frutt-append({ "Kiet", “watermeton’]) fruttcextend({* Cherry's Banana") fruit renove( Banana") ruse ‘rute-por() ‘rute-pop(2) Frutt (U) THe + operator works tke the extend method, except that it retums a new list a frustoc ID: 689693 Feutt (U) Other operators and methods tell how long a sis, whether an element is inthe Ist, andi so, where or how often itis found. ren(Frutt) fruit. append(‘Apple") ‘apple’ An frutt “Cranbercy’ not in fruse srutt.count( Apple") frutt.index<"Apple") _# Careful--con cause an error fruit.index(apple", 1) (U) List Comprehension (U) Great effort has been to make lists easy to work with. One of the most common uses of als is to iterate over its elements with a for loop, storing ofthe results ofeach iteration in a new ist. Pytion removes the repetitive boilerplate code from this type of procedure with list ‘comprehensions. They're best learned by example: [3 for $ sn range(s0)) [3002 for An range(29)] Ua, a2, for 4 in range(0)) (1a, a2, 1099) for 4 am range(ie) #4 x 2]. # conditionalst [{ej for 4 tn “abede"] for j in ‘ay2"] ——# nesting! (U) Sorting and Reorderingoc ID: 6589693 {8} Soting is another extremely common operation on lists. Wall covert in greater detail later, but here we cover the most basic builtin ways of sorting. The sorted function works on more than just List, but aways retums a new ist wih the same contents asthe orginal in sorted order. ‘There's also. sort method on Lists that performs an in-place sot. frult.renove({'Kiwi’, ‘Watermelon']) # can’t compare List with str sorted fruit = sorted(fruit) sorted fruit == fruit Frutt.sore() sorted fruit —~ fruit (U) Reversing the order ofa lists similar, with abuit-in reversed function and an in-place reverse method for ist s. The reversed function retums an iterator, which must be converted back into a list expbcily To sor something in reverse, you could combine the reversed and the sorted methods, but you should use the optional reverse argument on the sorted and sort functions. fruit = List(reversed¢Fruit)) fruit reverse() rfruit == frust sorted(r_frutt, reverse-True) (U) Tuples (U)Much tke a 14st, tuple is an ordered sequence of zero or more objects of any ype. They can be constructed by puting a comma- separated list of tems inside parentheses ( ) , or even by assigning a comma-separated list o a variable with no delimiters at al. Parentheses are heavily overloaded-they also indicate function calls and mathematical order of operations-so defning @ one-element tuple i tricky: the one element must be followed by a comma. Because @ tuple is immutable, it won't have any ofthe methods that change lists, ke append oF sort 2 (3, 2, “first and second") rena sorted(a) a index(2) ‘count (2)oc ID: 6559603 bert, (2, "3 type(b) tuple = 12", cra» «tuple dra © C8") d.ewple = (ds) draw tuple (U) Interlude: Index and Slice Notation (U) For the ordered containers 1st and tuple, as wel as for other ordered types lke st ings, it's offen useful to retrieve or change just one clement ora subset ofthe elements. Infexand slice notation are available to help with ths. Indexes in Python always start at 0. Wel start outwith 2 ew ist and work by example: aninals = ['tiger’, ‘monkey’, ‘cat", ‘dog', ‘horse’, ‘elephant*] aninals(2) ‘aninals(1] = “chimpanzee! ‘aninals[1:3] ‘aninals[3] in aniaals{2:5) animals[:3]—# starts at beginning animals[s:] goes to the end aninals[-2:] aninals(2:6:2] _# uses the optional step parameter animals[::-1] == List(reversed(aninals))Do ID: 6559603 {0} Because slicing retums a new lst and not just @ view on the list, Itcan be used to make a copy (technically a shallow copy}: sne_aninals = animals 4iferent_animals = aninals(:] sane_aninals[0] = ‘Lion’ aninals(0) ifferent_animals[2] = ‘Leopard’ Ai¢ferent_aninals[0) =~ aninals{2] (U) Dictionaries (U)A dict isa container that associates keys with values. The keys of a can be any type. (U) The dictionary construction shortcut uses curly braces ( ) with a colon : between keys and values (e.g. my_dict = (key: value, Keyl values) ) Alternate constructors are availabe using the dict Keyword. Values can be added, changed, or retrieved using index notation ‘with keys instead of index numbers, Some of the operators, functions, and methods that work on sequences also work wit dictionaries. bugs = {"ant": 10, “praying mantis": 0) {ct must be unique, and only immutable objects can be keys. Values bugst fly") = 5 ugs-update({"spider': 1}) Like extend et bugs spider") ‘ray’ in tugs 5 dn bugs bugs "Aly" (U) Dictionaries have several addtional methods speciicto ther structure, Methods that retum ists, Ike itens keys , and values , are not {Quaranteed to do so in any particular order, but may be in consistent order if no modifications ae made to the dictionary in between the calls. ‘The get method is often preferable to index notation because it does not raise an error when the requested key isnot found; instead, it retums None by default, ora defaut value that Is passed as a second argument.oe ID: 689603 bugs. ttens() # List of tuples bugs keys) bugs values) bugs. get") bugs. get( spider") bugs. getC"spier", 4) bugs-clear() ues (U) Sets and Frozensets (U)A set isa container that can only hold unique objects. Adding something that's already there will do nothing (but cause no error). Elements of ‘a sel must be immutable ike keys ina dictionary). The set and frazenset constructors take any iterable as an argument, whether it's ‘a List , tuple, or otherwise. Cuty braces around a ist of comma-separated values can be used in Python 2:7 and later asa shortcut constructor, but that could cause confusion wth the ict shortcut, Two sets are equalf they contain the same items, regardless of order. manbers = et( (1015 101413130303431202,203,344]) letters = set("ThequickBrownFoxtunpedoverTheL.azyDog" Lower()) 2 aice sre umbers = {05 25 34 4s Sh set runbers. 262) runbers. d6(3) roners. pdate((3, 4, 72) rnbers. 990) 1 could be anything runbers.renove(?)Bo ID; 6689693. ronbers.discard(7) 4 no error (U)A tozen setis constructed in a similar way; the ony ference isin the mutability. This makes frozen sets sultable as dictionary keys, but frozen ‘sets are uncommon, 2 Frozenset((1,1,21,2,3,3,3,3,32,2,2,3,3,4]) {(U) Sets adopt the notation of bitwise operators for set operations lke union intersection, and symmetric diference, This is similar to how the + operator is used for concatenating 1sst s and tuple s. house pets = {"dog', “cat, "fish") farmaninals = {cow's ‘sheep’, ‘pig's “dog's ‘eat"} hovse_pets & farmaninals # intersection house pets | faraaninals union house_pets * faraninals # symetric aifference house_pets - farmaninals # osymetric difference (U) There are verbose set methods that do the same ting, but wth two important iflerence: they accept List 6, tuple , and other terables as rugmens, and can be used to update the set n place. though there are methods corresponding to all the set operators, we give only a few examples. farm_aninal_List = List(farm animals) * 2 house_pets.intersection( fare aninal_ist) house pets. union( farm aninal_list) house_pets. intersection update(farm_animal_list) (U) Comparison of ses is similar: operators can be used to compare two sets, while methods can be used to compare sets with other iterabes, Unlike numbers or strings, sets are often incomparable. house pets = ("dog’, "cat", “Fish') farm_aninals > house pets house pets < farm_aninalsoc ID: 6689693, house pets. intersection update (farm aninals) Farm_aninals > house pets house pets. ‘ssubset(fare_aninal_list) (U) Coda: More Built-In Functions (U) We've seen how some built-in functions operate on one or two ofthese container types, bu all ofthe following can be applied to any container, ‘although they probably wort always wrk; that depends on the contents ofthe container. There are some caveats: + (U) When passed a dictionary as an argument, these functions look tthe keys ofthe dictionary, not the values. + (U) The any and 311 functions use the boolean context of the valves of the container, @.g. © is False and non-zero numbers are True , land all stings are Tove except forthe empty string", whichis False + (U) The sum function only works when the contents of the container are numbers. generic container = farm animals or bugs, annals, ete ali generic_container) any(generic_container) “pig? in genertc_container rot An generic container Len(generic_container) snax(generic_container) snin(generic_container) sum(L1, 2, 3, 4, 51) Lesson Exercises Exercise 1 (Euler's multiples of 3 and 5 problem)ee ID: 4689693, Ifve ist all the natural numbers below 10 that are multiples of 3 or 5, we get 3,5, 6 and 9. The sum of these multiples is 2. Find the sum of athe muliples of 3 or 5 below 1000. Exercise 2 ‘Wote a function that takes a ist as a parameter and retums a second lst composed of any objects that appear more than once in the orginal list + duplicates(1,2.36,73,4,56)) should return [3,6] + what should dupicates(Tcow pig goat: hose] return? Exercise 3 \Wrto a function thet takes a portion mark as input and retums the ful classification + convert_lassification(U/FOUO) should return 'UNCLASSIFIED/FOR OFICIAL USE ONLY" + convert_classification(S/REL TO USA, FVEY' should return SECRETIREL TO USA, FEV" UNCLASSIFIEDoe ID: 689693 Lesson 05: File Input and Output pene upate almost 2 years ago by [Tin chi sz ated ee (U) Lesson 05: File Input and Output UNCLASSIFIED (U) Introduction: Getting Dangerous (U) As you probably already know, input and output isa core to! in algorithm development and reading ftom and wring to files is one ofthe most ‘common forms. Lo’ jump right in justo Soe how easy itis to write a fe, myfile = open(“data.txt", ‘w") myfile.write("I an writing data to ny file") yfile.close() (U) And there you have it You can write data to fles in Python. By the way, the variables you put into that open command are the filename (asa sring-do not forget the path) and the fle mode. Here we are wring the fle, as indicated by the x’ as the second argument 1 the open function. (U)Lettear apart what we actually did ‘open(data.tet', "w") (U) This actully returs something caled a fle object. Let's name it (U) EEE) Opening a fie that already esas for wring will erase the original fle. myfile = open(‘data-tet", ‘w")oo ID: s6s9693, (U) Now we have a variable to this fle object, which was opened in write made. Let try to wre to the fe: ryflle.write("r an writing data to my File") ryfile.read() # 0ops...notice the error nyfile.close() * Guess what thot ata (U) There are only a few file modes which we need to use. You have seen "x (wring), The others ‘are 'r* (reading), "3° (eppencing), ‘r+* (reading and wring), and s* (binary mode). myfile = open(‘data-txt', ‘") myfile.read() mflle.wrtte("r am writing more data to my file") Oops again. .check our mode data ~ myfile.read() iydata 1 HEY! Mere did the dota 90. myfite.close() # don’t be 0 piggy (U) Aco! way o use contents of a fi in a black is withthe ith command, Formally, this is called a context manager. Informally, it ensures that the fle is closed when the block ends. saith open(*data.txt") as F: print(.read()) (U) Using sien is a good idea buts usualy not absolutly necessary. Python tries to close files once they are no longer needed. Having fles open | not usually a problem, unless you try to open a large number al at once (@.9. inside a oop). (U) Reading Lines From Files (U) Here are some of the other useful methods for le objects: Lines _File ~ open(fewlines.txt", ‘w') Lnes_file.writelines(“First\n") Lines_File.writelines({"second\n", “thtrd\n"]) Lines_file.close()‘open(*Fewlines.txt", 0") 1ines_File.read2ine() 1ines_file. read2ine() nes_Fie.read2ine() nes_File.readzine() (U) And make sure the fe is closed before opening it up again in the next cell ines_File. close() (U)Aternatety ines = open("Feulines.txt", ‘r*).readlines() # Note the plurality ines (UE both reas and resaine(s) have optional size arguments that iit how much s read. For resdline(s) this may return incomplete lines. (U) But what ithe file is very long and | dont need or want to read all of them at once. ie objects behave as ther own iterator. Lines_File = open( "Fevines.txt", 'r") for line in 1ines File: rine 2ine) ‘The below syntax is avery common formula for reading through files. Use the with keyword to make sure everything goes smoothly. Loop through the file one line ata time, because often our les have one record to a line. And do something with each line. with open("feulines.txt*) as my_flle: for Line in my file: print(Line.strip()) # The strip function removes newlines and whitespace fran the start and fintsh “The fle was closed upon exiting the with block. (U) Moving Around With tell and seekoe ID; 6649603 (0) The tei method returns the current posiion ofthe cursor within the file. The seek command ses the current postion of the cursor within the fle, Inputfile = open(‘data.txt", 'r") Anputfie.t011() Anputfite.read(s) Anputfie.te110) Anputfite. seek(0) input file. read() (U) File-Like objects (U) There are other times when you really need to have data in a fie (because another function requires it be read from a fle perhaps). But why waste ime and disk space if you already have the data in memory? (U) A very useful module to make a sting into a fle-ike object is called Stringro . This wil ake a sting and give it fle methods lke read and write snport 0 systeingfile = f0.string10() 4 For handing bytes, use to.Bytesto systringftle.welte("Ts 4s ay datal) # he Just wrote f0 the object, not a Fllehandle systringttte.read() 4 cursor {s at the end ystringtite.seek(0) systringtite.reaa() nevstringfile = Jo.stringlo("wy data") # The cursor will autonaticatly be set to @ (U) Now les pretend we have a function that expects to read data from a file before it operates oni. This sometimes happens when using ibrary functions. def iprintaatac) rint(F.read())torintdata(‘mdata’) rrr "y 40 « f0.Stringio( ‘nydata*) Sprintdata(ey so) # vay Lesson Exercises Get the data ‘Copy sonnet from htios.Jur.nsa.ic.qavlvixGam and paste into sonnet. txt. Exercise 1 Write a function called 1 input file with only the ie_capitalize() that takes an input fle name and an output fle name, then writes each word from the input fi Wy ‘rst letter capitalized tothe output file. Remove all punctuation except apostrophe. capitelize("sonnet txt’, ‘sonnet_caps.txt") => capitalized words written to sonnet cap Exercise 2 ‘counts for each word. Remove all punctuation {il @ function called fle_word_ count) that takes a fle name and retums a dictionary containing the counts for ‘except apostrophe. Lowercase all words. File_word_count(“sonnet txt") => { "at's 4, tne: 2, oa} Extra Credit Wie the counts dictionary toa file, one key-value per line. UNCLASSIFIEDoc ID: 6689693 Lesson 06: Development Environment and Tooling Created over 3 years ago by [Tin cove saat sya OO Oo es (U) Lesson 06: Development Environment and Tooting (U) Package Management (U) The Problem: Python ha leaves something to be desired + Python doesn'thave a classpath and unless you are root , you cant intall new packages forthe whole system, + How do you share a script with someone else when you dont know what packages are installed on ther systom? () G)rP.b, 86=3E atteres included" philosophy-it has a comprehensive standard library, but by default, using other packages + Sometimes you have to use Project A, which relies on a package that requires awesome-package v.11, but youre writing Project B and want fo use some features that are new in awesome-package v.20? + The bestin-class package manager isn in the Python standard ibrar. (U) The Solution: virtualenv (U) The vintuaieny package creates virtual environments, i. isolated spaces containing ther own Python instances. It provides a uly script ‘that manipulates your environment to activate your environment of choice. (U) Its already installed and available on the class VM. The -p flag indicates which Python executable to use asthe base for the vitual ‘environmentDoe 1D: 6659693 (3.35 wh ctaatony neweny p_Jusr/locat/bin/nython new python executable Installing Setuptoots Installing Pip.» Sta use[iocatoineython — = NEMENY/n/ pyERon C—O i (U) The virtuateny package can be cowoloadeand fin gs a Sip o create a vitual environment based on any recent Python installation. A Viual environment has the package manager hip preinstalled, which can be hooked ito the internal mor of the Python Package Index ()2!) by exporting the correct agdress'to the, 0" INDEX URL environment variable: (32s cc srt ne Um Ieap/Ttbrantiacep area. 1.40/97 (F238 pytnon Python 2-7-5 (default, Nov 6 2013, 10:23:48) [Gee 4.8.7 20120313 (Red Hat 4.4.7-3)] on Linux? Type "help", "copyright", “credits” or “License” for nore information. Amport requests Traceback (ost recent call last): File “estdin>", Line 1, in Inportrror: No module naned requests ext,oe 1D: 6589593 (323s spurge nenenv/oinyactivate (Fis spires WORN LEESEE: Downloading/onpacking requests Downloading requests-2.0.0.tar.gr (362K8): 362K8 downloaded Running setup.py ega_info for package requests Installing collected packages: requests Running setup.py install for requests ce sucessfully installed rests + Cleaning vp. (0) 2 yon Python 2.775 (Gefwolt, Nov. 6 2013, 10:23:48) [ce 4.4.7 20120913 (Red Hat 4.4.7-3)] 09 nun? Type helps “copyright, “ered” or “License” for sore infizsion. Amport requests vrequests.__version_ "2.0.0" Amport sys sys.path C', ‘none —Sieweue7io/pytnon27.249", */mone{——ueuenv/si0/oytnonz.7°, * ‘mone newenv/1sb/pytiona.7/pat- Linixa', “home f———NEWENV/11b/pythok2.7/14b-tk", "7none/[——/NEWENY/14b/python2.7/14b-084", ‘hone /[—__NENENV/14 b/python2.7/1ib-dynlbad', */usr/lecalylib/python2.7", '/usr/local/1ib/python2.7/plat-Linux2", ‘/usr/local/1ib/python2.7/11b-t 1)" Pnoe/P NEMO Ab python. 7/site-packages} ext) (W000 =F “5 tp Frere woetrefeces?oc ID: 6559693 ‘Now we have a place to install custom code and a way to share it! + Develop code inside ~/neven/1ib/python2.7/site-packages + Capture installed packages with pip freeze >> requirenents.txt and install them to anew virtualenv with pip snstall requirenents.xt (U) The Ultimate Package (U) 8ython is an alternative interactive shell for Python with los of cool features, among which are: ‘a web-based notebook interface with easy-o-share fies, and + distributed computing (dont ask about tis) (U) To get state:oe 1: 6685698 (nese ——, Domnloading Spython-12.6 Successfully installed ipython Python 2.7.5 (Gefault, Nov 6 2013, 10:23:48) Type “copyright”, “credits” or “License” for more information Python 1.1.0 == An enhanced Interactive Python 2 ~> Introduction and overview of Python's features. quickret -> Quick reference help -> Python's own help syste. object? => Details about ‘object", use ‘object??* for extra details. an (1): Is BASE3/ Hello Moré. html Hello World. ‘pyn NEMENV/ na}: hist as In[3}: smport 08 Inf}: os.path #press tab foes os.path os-pathconf os,pjtiicont_paes os.pathsep In (4): os.path eae (U) Touse the web interface, you hadg 1 install sjplemental packages: (WEVENY =F 245 in inetd pyena tornado Jinja2 pygnents (Nee) ————T = py notebook --no-nathjex (uss mo mae potages a ret o et awasoe nin rahies I (NEMENV) [LJ ~1 pip install matplotliboe ID: 6559693 Lesson 07: Object Orienteering: Using Classes Updated. 9 months ago by 23 me (U) Introduction to classes, objects, and inheritance in Python UNCLASSIFIED (U) Introduction (U) From the name oft you can see that object-oriented programming is oczing with abstraction and complication. Take heart: there's no need to ‘ear or avoid object-oriented programming in Python! Is just another easy-to-use, flexible, and dynamic tool in the deep toolbox that Python makes avaiable. In fac, we've been using objects and object oriented concepts ever since the fist ine of Python code that we wrote, so ifs already {amili. In this lesson, well think more deeply about what itis that we've been doing all along, and how we can take advantage ofthese ideas. (U) Consider, for example, the diference between a funetion and a method: name = hark™ en(nane) function rame-upper() # method (U) Inthis example, ane is an instance of the st type. nother words, mane is an object ofthat ype. An objects just a convenient wrapper ‘around a combination of some data and functionality related to that data, embodied in methods. Until now, youve probably thought of ‘every str justin terms of its data, ie. the literal string “hark that was used to assign the variable. The methods that work with ane were defined just once, ina class definition, and apply to every string thatis ever created. Methods are actually the same thing as functions that lve inside a cass instead of outside it. (Tis paragraph probably sill seems really confusing, Ty rereading it atthe end ofthe lesson!)a ID: 659693 (U) Your First class (U) Just asthe keyword def is used to define functions, the Keyword class is used to define a type object that will generate a new kind of objec, hich you get to name! As an ongoing example, well work wih a class thal wel choose to name. Person = ‘lass Person(object) pass ‘ype(Person) *ype(Person) == typecint) nebedy ~ Person) ‘ype(nabody) (U) Atfist, the Person class doesn't do much, because i's totaly empty! This isnt as useless as it seems, because, just Ike everything else in Python, classes and their objects are dynamic. The (sbject) afer Person is not a function call; here it names the parent class. Even though the Person class looks boring, the fundamentals are there: + the Person clas is ust as much ofa class as int or any other bullin, + we can make an instance by using the class name as a constructor function, and + the type of the instance nobody Is Person just ike type(2) I int (U) Since that's about all we can do, le’ start over, and wrap some data and functionality into the Person class Person(object): Species = “Hono sapiens” ef tatk(sel®): ‘return "Hello there, how are you?” nobody = Person() nobody. species nobody. talk() (U) Its very important to give any method (:. function defined inthe class) atleast one argument, which is almost always called self. This is because internally Python transates nobody. talk) into something ike Person.talk(nobady) (U) Lets experiment with the Person class and its objects and do things like re-assigning other data atrbutes. sonebody = Person()Bo ID: 689693 sonebody.species = ‘Homo internetus’ sonebody name = “Mark” robody. species Person. species = "Unknown" nobody. spectes somebody. species erson.nane = “Unknoxn” nobody.nane sonebody rane ‘det soncbody.nane sonebody name {(U) Attrough we could add a nase to each instance just after creating i, one ata time, wouldnt tbe nice to assign instance-specifc attributes tke that when the object is frst constructed? The __init_ function lets us do that. Except fr the funny underscores in the name, i just an ordinary function; we can even give it default arguments. lass Person(object): Species = “Hono sapiens” def _init_(selF, nane-"Unknom", age-18) Self name = rane self.age - age def talk(seif): return “Hello, my nane is ().*format (self nane) mark = Person("Mark”, 33) everic_voter = Person() generic worker = Person(age-<1) seneric worker. ageoc ID; 6559693 generic worker. name (U) In Python, it int unusual to access attributes of an object directly, unike some languages (e.9 Java), where that is considered poor form and ‘everything is done tough getter and seer methods. This is because in Python, attributes can be added and removed at any time, so the getters and setters might be useless by the time that you want to use ther, mark, favorite color » “green del generic_worker rane generic worker, name (U) One potential downside is that Python has no real equivalent of private data and methods; everyone can see everything. There is a polite convention: other developers are supposed to treat an atrbute as private fits name stats with a single underscore (_). And there is also 8 trick: names that start with two underscores ( _) are mangled to make them harder to access. (U)The __init__ method is just one of many that can help your class behave lke a fullledged built-in Python object. To control how your object 's printed, implement __str__, and to control how itlooks as an output from the interactive intorprter, implement _repr__ Ths ime, we wont lar rom scratch; weil add these dynamically ef person_str(self): return "Mane: {8}, Age: {2)".format(self.nane, self. age) Person._str__ = person_str ‘def person_repr (self) return "Person({@) {2))* format(self.nane, self age) Person.__repr__ = person_repr print (mark) # which special method does print use? snark which special method does Jupyter use to auto-print? (U) Take a minute to think about what just happened: ‘+ We added methods toa class after making a bunch of object, but every object in that class was immediately able to use that method. + Because they were special methods, we could immediately use builtin Python functions (Ike str )on those objects. (U)Be careful when implementing special methods. For instance, you might want the default sort of the Person class tobe based on age. The special method __1¢_(self,other) willbe used by Python in place ofthe builtin 1¢ function, even for sorting. (Python 2 uses cep instead.) Even though its easy, this is problematic because it makes objects appear tobe equal when they are just ofthe same age!Doe ID: 6589693 ‘def person_ea(self, other): return self.age == other. age Person._eq_= person_ca bob = Person("t 33) bob == mark (U) na situation ike ths, it might be beter to implement a subset ofthe rich comparison methods, maybe just it _ and _ gt more complicated __eq__ function that is capable of uniquely dentying all the objects you wil ever creat. (U) Wile we've shown examples of adding methods to class after the fact, note that itis rarely actually done that way in practice. Here we did ‘that just for convenience of not having fo re-define the class every lime we wanted to create a new method. Normally you would just define all lass ‘methods under the class itself If we were to do so withthe _str_, _+ 1nd _eq_ methods for the Person class above, the class would lke the below: lass Person(object): species = “Hono sapiens” def _init_(self, name-"Unknown", age=18): Gelf.nane = name self age ~ age ef talk(sel#) return “Hello, ay nane 1s ().".format (self. name) def _str_(self)) ‘return "Name: {0}, Age: {1)".format(self.nane, self.age) ef _repr_(self): Feturn "Person( "(0)", (1))*format(self.nane, self.age) def _eq_(self, other): ‘eturn self. age == other.age (U) Inheritance (U) There are many types of people, and each type could be represented by is own class. It would be a pain ifwe had to reimplement the fundamental Person traits in each new class. Thankfuly, inheritance gives us a way o avoid that. We've already seen how it Works: Person inherits from (ois a subclass of) the object class. However, any lass can be inherited from (0. have descendants),Doc ID: 6689693 class student (Person): bedtime = "Midnight” (def do_honework (self): Anport tine print ("T need to work.) ine. s1e¢p(5) print("Oid 1 Just fall asleep?) tyler = student(“Tyler", 19) tyler. species tyler-talk() tyler. do_honework() (U) An objec from the subclass has all the properties ofthe parent cass, along with any additions from its own class definition, You can sil easy to ‘override behavior fom the parent clas easily-just create a method with the same name inthe subclass. Using the parent class's behavior inthe child clas is ticy, but fun, because you have to use the super function, ‘class Employee(Person): ef talk(sel¥): talk_str = super(Eaployee, self).talk() return talk str +" T work for {}*.format (self. eaployer) fred = Employee("Fred Flintstone”, 55) fred-enployer = "Slate Rock and Gravel Company” fred. talk() (U) The syntax heres strange a frst. The super function takes a class (Le. a type ) 8s frst argument, and an object descended from that lass as its second argument. The object has a chain of ancestor classes. For fred that chain is Employee, Person, object] ‘The super function goes through that chain and returns the lass thats afferthe one passed asthe functions fist argument. ‘Therefore, super can be used to skip up the chain, passing modifications made in intermediate classes, (U)As a second, more common (but more complicated) example, i's often useful to add addtional properties to subclass objects in the constructor.oc ID: 6589693 class Employee(Person) def _init_(self, nane, age, enployer): Super(Employee, self). init_(nane, age) self.enployer ~ exployer ef talk(sel#): talk_str = super(Eaployee, self).talk() return talk.ste +" T work for {)*.Format(self. employer) fred = Eaployee("Fred Flintstone", 55, "Slate Rock and Gravel Company") fred. talk() (U)A class in Python can have more than one listed ancestor (which is sometimes called polymorphism). We wont go into great detail here, aside from pointing ou that exsis and is powerful ut complicate. ‘lass Studenttaployee(student, Employee): pass ann = studentEmployee("ann", 58, "Family Services”) ann.talk() bLLL = Studentemployee("bi1I", 20) # what happens here? why? (U) Lesson Exercises (U) Exercise 1 (U) White a Query class that has the folowing attributes: + classification + justiteation + selector (U) Provide default value foreach attribute (consider using one). Make itso that when you print, you can display al ofthe attributes and their ‘values nicely. # your class definition here (U) Afterwards, something ike this should work:oc ID 6689693 query! = Query("SSASEHREL IGG SISK, “Primary email address of Zendian diplonat”, “ileonafstato.gov.24") print (query) (U) Exercise 2 (U) Make a RangedQuery class that inerits from Query and has the adltional attributes: + begin date + end date (U) For now, just make the dates ofthe form YYYY-MIM.DD. Dont worry about date formating or eror checking for now. Wel tak about the satetine module and exception handling iter. (U) Provide defaults fr these atibutes, Make sure you incorporate the Query class's ializer into the RangedQueryinalzer. Ensure the new ‘lass can also be printed nicely. 1 your class definition here (0) Atgewards this shoul woe: query2 = RangedQuery(SeiiGEtiRetHe-Noh FHM, “Primary IP address of Zendian diplonat", "18.254.18.162", "2016-12-01", "261 print (query2) (U) Exercise 3 (U) Change the Query class to accept alist of selectors rather than a single selector. Make sure you can sil rit everthing OK. UNCLASSIFIEDee: 6589603 Lesson 07: Supplement est ‘months ago by[————Tia CONE 3824 ce (U) Supplement to lesson 07 based on exercises from previous lectures. ‘You may have written function lke this to check ifn tem is in your grocery stand print something snarky if ts not: ef Sn my ist (Aten): ay list = ['apples* AF item in my List: return "ot St!" else: ‘return ‘Nope! * ‘nt2k*, "butter', ‘orange Juice’) in_ny_Ase( apples") in_ay_List("ehocel: ) ‘But what if really wanted chocolate tobe on my ist? | would have to rewrite my function. I had writen a class instead ofa function, | would be ‘able to change my list. class ny List (object): ny list = ['apples' ,’milk',"butter’, ‘orange Jutce"] def in_ny List (self, item): AF item in self ay List: return ‘Got itt” ese: return ‘Nope! *oo ID: 6689693 ecenber = Hy 1ist() december. in my list(‘chocolate") ecenber.ay list decenber my list +['chocolate"] ecenber.in my List(‘chocolate") Now Ihave a nice template for grocery ists and grocery list behavior jan = my tist( decenber-ay list jon.my ist ‘This ont helpful: print (decenber) So.we overwrite the _tr__function we inherted from object: lass Wy List(object): ry list = [‘apples", ‘milk’, ‘butter’, ‘orange juice’) def _str_(eelf): ‘return "hy List: ()° format(*, *Join(self.ay_1ist)) def _repr_(self): ‘return self._str_() def in_ay_List (self, tem): Af Stem An self ay List: return “Got iti” else: return “Nope! " ecenber = Hy_List() print (decenber) december Maybe | also want to be more easly testify favorite snack ison thelist...oe ID: 6689653, lass My. list(object): ay list = [‘apples', ’at2k*, ‘butter’, ‘orange Juice”) ef _init_(selF, snack-' chocolate’): Belt snack = snack def _str_(self): return "Wy List: ()° format(", *-Join(self.ay_list)) (def Sn_my_1ist (self, stem) AF Stem in self my List: ‘return ‘Got Sti" ase: ‘return ‘Nope! * (def snack check(sel#): return self.snack dn self.9y list ‘my favorite snack ts chocolate... But tn Jonuary I'm going to pretend it's oranges jan = My_List(‘apples") an. snack check() ‘s8ut in February, I'm back to the defoult feb = My 1st) ‘Feb. snack_check() ‘About that object. ‘ir(object) These are all the things you inher by subclassing object. lass caps_List hy list): ‘def in-my_List(self, item): response = super(caps_list, return response.upper() AF) Anny AAst( item) shouty = caps 1ist() shouty-in_my_List("chocolate’) ir(caps aust) You can also call the super class directly, ike so:oe ID: 6589693 lass caps_List (my list): def in_ny_List(self, item): "© but you stilt have to pass self response = Hy_list.in_ny_list(self, item) ‘return response.upper() shouty = caps 1ist() shouty.in_ay_list( chocolate’) ‘Super actually assumes the correct things... Most of the time. lass caps_List (my_list): def in ay List(self, stem): ‘response ~ super().in_ay_list(item) ‘return response.upper() shouty = caps_14st() shouty.in_ay_list( chocolate’) hetp¢super)Doc ID: 6689653 Lesson 08: Modules, Namespaces, and Packages Upaigd rr 2 er apo bp — ici aera = (U#FUO) A lesson on Python modules, namespaces, and packages for COMP3321. UNCLASSIFIEDHFOR-OFFIGIAL USE-ONEY (U) Modules, Namespaces, and Packages (U) We have already been using modules quite a bit — every te we've run Inport, in fact. But what is a module, exactly? (U) Motivation (U) When working in Jupyer, you don't have to worry about your code disappearing when you exit. You can save the notebook and share it with ‘others. A Jupyter notebook kind of behaves like a python seript: a tex fe containing Python source code. You can give that fle tothe python Interpreter on the command line and execute al the code inthe fle (kind of like "Run Alin a Jupyter notebook}: $ python awesome.py (U) There are a few significant liitations to sharing code in Jupyter notebooks, though: 4. what if you want to share with somebody who has python installed but not Jupyter? 2 what ityou want o share part f the code with others (or reuse part of it yourself}?oc ID: 689603 "Swati you're writing a large, complex program? (U) Alot these do have native solutions in Jupyter: 4. convert the notebook o a script (Fle > Download as > Python) 2 copy-paste..? 3. make'a big, messy notebook... (U) but they got unwieldy fast. This is where modules come in, (U) Modules (U)Atits most basic, a module in Python is really just another name fora scrip. I's just a fle containing Python defritions and statements. The flename is the module's name followed by a py extension. Typically, though, we don't run modules directly ~ we import their defnions into our ‘own code and use them thee. Modules enable us to write modular code by organizing our program into logical units and putting those units in ‘separate files. We can then share and reuse those fle indvidualy as pars of other programs. (U) Standard Modules (U) Python ships with a library of standard modules, so you can get prety far without writing your own. We've Seen some ofthese modules already, ‘and much of next week willbe devoted to learning more about useful oes. They are documented in full detain the Pyition Standard Liorary ‘ference. (U) An awesome example (U) To understand modules better let's make our own. This wil put some Python code in a fle called avesone.py inthe current directory clase Avesome object): def _init_(self, avesone_thing) Self.thing = avesone_thing def _str_(self): return "{@.thing) 4s avesone!!-foraat(self) a = fwesone( Everything”) print(a) with open(‘awesone.py', "w") as #2 F.weite(contents)1: 6689685, {(G)Now you can run python awesone.py on the command line as a Python script. (U) Using modules: import (U) You can also import swesone.py here as a module: Anport avesone (U) Note that you leave out the file extension when you impor it. Python knows to lok fora le in your path called awesome. py (U) The frst time you impor the module, Python executes the code inside it. Any defined functions, classes, ec. willbe availabe for use. But notice ‘hat happens when you try to impor it again: Anport avesone (U) ts assumed thatthe other statements (e.g variable assignments, print) are there to help initialize the module. That's why the module is only run once. Ifyou try to import the same module twice, Python will not re-run the code ~ it wil refer back to the already-imported version. This is helpful when you import multiple modules that in turn impor the same module. (U) However, what the module changed since you last imported it and you really want to do want to resimport i? contents = class Avesone object): def _init_(self, awesone_thing): Self. thing = avesome thing ef _str_(self): return “{@.thing) 45 awesonel!!*format(sel#) ef cool(group): return “Everything is cool when you're part of (@)".Format(group) 1 = Awesone( Everything”) prine(a) vith open(‘awesone.py', 'W") a5 £2 F.wrive(contents) (U) You can bring in the new version with the help ofthe nport1sb- module import inportlib {nportiib.reload(awesone)Doc ID: 689603 (UY'calling the module's code (U) The main point of importing @ module isso you can use its defined functions, classes, constants, etc. By default, we access things defined in ‘the awesone module by prefixng them with the module's name. print (awesone.Anesone("A Nobel prize") mesome cool" tean") print (awesone. a) (U) What ifwe get ted of wing avesone all the time? We have a few options. (U) Using modules: import __ as (U) Fist, we can pick a nickname forthe module: import avesone as ave print (owe. Awesone("A book of Greek antiquities”)) awe.cool("the Python developer comunity”) print (awe. a) (U) Using modules: from ___ import ___ (U) Second, we can import specific things from the. suesone module into the curent namespace: from avesone import coo} cool("this class") print(Awesone("A piece of string")) # will this work? print(a) # will thes work? (U) Get everything: from ___ import * (U) Final, if you really want to import everything from the module into the curent namespace, you can do thisDoc ID: 6589693 ‘fron awesone import * 7 BE CAREFUL (U) Now you can re-run the cals above and get them to work (U) Why might you need to be careful with this method? 1 what {F you had defined thts prion to import? ef 001) return "Something important 4s pretty cool” impart «the ‘arabes are imported into your current namespace. Think ofa namespace as a super-variable that contains references to lots of eer variables, or ‘8 a superclass that can contain dat, functons, and classes. (U)Ater import, a modules dynamic ike any Python objec: for example, the oeloss function takes @ module as an argument, and you can add data and methods tothe module afer you've Imports i (but they woa' perit beyond the Iftime of your script or session). Amport my_funcs def si1ly_fune(x): return "Silly ()!*format(x) re sflly_fune = stlly_fune we stlly_fune(“hark™) Silly Markt (U)Incontrast, the from cmodule> ‘eport Command adds the function tothe current namespace, (U) Preventing Excess Output: The Magic of __main__ (0) Suppose you havea script that does something awesome, called avesonepy class Aesone object) def _ init__(self, awesone_thing): Self. thing = awesone_thing def _str_(seif) Feturn “{@.thing) 4s AWESOME.” Format( self) 12 = Awesome( "BASE Jumping”) print(a) (U) This can be executed from the command line oroported:* ns Cooma re (vw) bythonoc ID; 6689693 mort avesone BASE Dumping 15 AWESOME Traceback (gost recent call last) File “estéin>", Line 1, in enodule> NaneError: mane “a {5 not defined ‘awesome. twesome object at @X7fa222a8b420> print (awesome. 2) BASE Jumping 1s AVESONE (U) You don't want that print statement to exeoute every time you import. Of equal importance, avesone.s Is probably extraneous within an Import. Lot's fix io get rid of those when you import the module, but keep them when you execute the script class Awesome object): def _init_(self, awesone_thing): Self thing = avesone thing def _str_(seif): Feturn "{@.thing} 4s AWESOHE.~.Format(self) 4 _name_ == *_ ain 1 = Avesome("BASE Jumping”) print(a) (U) We can do even beter. There are some situation, e.g. profiling or testing, where we would want to impor the module, then look at what would happen if we run its a script To enable that, move the main functionalty into a function called main()oc ID: 6689695 lass Avesone object): def _init_(self, avesone thing) Selfthing = avesone thing def _str_(self): Feturn "{@.thing) 4s ANESOHE.”.Format(sel#) def main): 1 = Auesome(“BASE Jumping") prine(a) $F names *_nain_ ain() (U) From Modules to Packages (U)A single Pytnon module corresponds oa file. Its not hard to imagine a situation where you have several related modules that you want to group together in the equivalent ofa folder; the Python term for this concept isa package. We make a package by + creating a folder + puting sripts/modules inside it + ding some Python Magic (which obviously willinvolve __ in some way, shape, or form) (U) For example, wel put ouesone.py in a package called feelings-later on, well add terrible.py and totally_rad.py The directory structure ie feelings/ J awesone.py fF _inte_py Tain py (U) The _snit_.py fle is REQUIRED; without it, Python won't identily this folder as a package. However, __main_py is optional but nice; if you have i, you can ype python feelings and the contents of main _.py willbe executed as a sript. (NB: Now you can postulate on what F Lane == "ain": i really doing (U) The _init_.py fle can contain commands. Much ke the _init_() function ofa class , the _init_py is executed immediately ‘after importing the package. One common use is to expose modules as package attributes; allthis takes is sport «nodule nane> in the package's init__.py Me,oe ID: 6589693 (U) Onward to the Whole World Pretty soon, youll want to share the feelings packages witha wider audience. There are thousands of people who want to do Auesone stuff, but " Mine 2 ‘symtaxérror: Snvalid syntax (U) Python could not parse what we were trying to do here (because we forgot our colon). I di, however, let us know where things stopped making ‘sense, Note the printed line with an ny arrow (“pointing to where Python thinks there isan issue (U) The statement syntaxtrror: invalid syntax Is an example ofa special exception called a SyntaxError. Its fay easy to Soe what happened here, and there is not much to do besides fing your typo. Other exceptions can be much more interesting, (U) There are many types of exceptions: Anport butttins # This will display @ Lot of output. 4 To wake it scrollable, select this cell and choose 4 Cell > Current Output > Toggle SeroLLing help(buittins) 4 Python 2 used to have this info in the “exceptions” module f python 2 moved it into “builtins” for consistency 48 50 for python 2, try this instead Anport exceptions ‘ir(exceptions) (U) bet we can make some ofthese happen. Infact, you probably already have recent 10 ef £0: ve import chris fie open( ‘data’, Fite. read()Do ID: 6589693 (U) Exception Handling (U) When exceptions might occur, the best course of action to to handle them and do something more useful than exit and print something to the soteen. Infact, sometimes exceptions can be very useful tools (e.g. Keybosrdinterrupt ). In Python, we handle exeptions with the try and except commands. (U) Here is how it works: 41. (U) Everything between the try and excest commands is executed, 2 (U) that produces no exception, the except block Is skipped and the program continues. 43 (U) fan exception occurs, te rest ofthe try Block ie skipped! 44 (U) tthe type of exception is named after the except Keyword, the code after the except command is executed. 5 (U) Otherwise, the execution stops and you have an unhandled exception. (U) Everything makes more sense with an example: ef F00: ty: ‘print (°I am going to convert the input to an integer") print (int(x)) except ValueError: ‘Print ("Sorry, Twas not able to convert that") Fo) #2) FC") (U) You can add mutiple Exception types tothe except command: except (Typetrror, Valuetrrer) (U) The keyword as Jets us grab the message from the error:Do ID: 6589693 ‘def be_careful(a, b): wy: print(Float(a)/#loat(b)) except (Valuetrror, TypeError, ZeroDivistonérror) as detail: print("Mandled Exception: “, detail) except: ‘Print “Unexpected error!*) ‘finaly: ‘rd (“THIS WELL ALWAYS RUN") be_careful(s0) be_careful(i (2421) be_careful(1,"two") be _careful(15"*400,1) ‘Aloat(. 128) (U) Weve also added the Finally command. twill always be executed, regardless of whether there was an exception or not, 80 t should be used {8a place to clean up anything let over from the try and except clauses, e., closing files that might stil be open. (U) Raising Exceptions (U) Sometimes, you wil want to cause an exception and let someone else handle it. This can be done wih the aise command. raise TypeError("You submitted the wrong type") (U) no builtin exception is suitable for what you want to raise, defining a new type of exception is as easy as creating @ new cass tha inherits from the exception type. ‘lass nypersonalerror( Exception pass raise MyPersonalérror("I am mighty. Hear my roar!)oe ID; 6589693 def tocater (ayLocation) AF (nylocation» factortal(s) xe ef Factorial(n): sr'neturn the factorial of n, an exact integer >= >>> [factortal(n) for # An range(6)] [4 1, 2 6 24, 120) >>> factorsal(30) 265252459812191058636208480000000 >>> factorsal(-4) Traceback (nost recent call last Valuetrror: must be >= @ Factorials of Floats are OK, but the float must be an exact integer: >>> factordal(30.1) ‘Traceback (nost recent call last): Valuaérror: n must be exact integer do» factorsal(3@.0) 265252859812191058636308¢80000000 Tt mist also not be ridiculously Lange: >>> factorial(1¢100) Traceback (nost recent call last): overFlovError: n too larg import math Af not n= raise ValueError(" must be >= AF nath.Floor(n) |= 1:oe ID: 6559693 raise Valuetrron(“n must be exact integer”) Af mri == ni # catch 2 volue Like 10200 raise OverflovError(n too large") result = 2 factor = 2 whkle factor 200: raise StopIteration for y in self.avoid multiples: Af self.x Ky = 8: self x break ores else: return self. ef _iter_(self): ‘return self stlent_Fizz_buzz ~ NonFactortterable(3, 5) [x for x in stlent_fz2_bur2] rnostly_prine = NonFactortterable(2, 3, 5, 7% 12, 13, 17, 19) partial_sun - @oc ID: 6559583 for x in nostly_prine: partial sum x partial_sum mostly prine = NonFactortterable(2, 3, 5, 7 11, 13, 17, 19) print (un(aostly_prine)) (U) tmay seem strange thatthe __‘eer__ method doesnt appear todo anything. This is because in some cases the terator for an objec should not be the same as the object itself. Covering such usage is beyond the scope of the course, (U) There is another way of implementing custom iterator: the __getiten_ method. This allows you to use the square bracket {] notation for ‘gottng data out of the object. However, you stil must remember fo raise a Stopiteration exception fort to work properly in for loops and ist comprehensions. Another iterator example Inthe below example, we create an iterator that retums the squares of numbers. Note that in the __next_ method, all we're doing is iterating our counter ( sei.» ) and returning the square ofthat counter number, as long as the counters not greater than the pre-defined limit (self.imit ) The while loop inthe previous example was speci o that use-case; we don’ actually need to implement any looping at alin _next._, a8 that's simply the method called for each iteration through a loop on our iterator. Here we're also implementing the _getiven__ method, which allows us to retrieve a value from the erator at a certain index location. This one. simply calls the terator using self. next untlit aries atthe desired index location, then retums that value.oc 1D: 6689693 class Squares (object def _init_(self, Linit-200): Seif tinit = Limit self.x= 0 ef __next_(sel4): welt 3 Af self.x > self.tinit: raise Stoptteration return (self.x-1)"*2 def __petitem_(self, 16x): Fintetottze counter to 0 self.x = 0 4€ not Asinstance(idx, int): raise Exception("Only integer index arguments are accepted!") Mitte selfx ¢ ddx: self._next_() return self x ef _iter_(sel): ‘return self Squares(1init=20) sy sauares (x for x An ay_squares} vy_squares{5} # since we set o Lintt of 20, we can’t access an index Location higher than that y_squares[25) (U) Benefits of Custom Iterators 4. (U) Cleaner code 2 (U) Abily to work wth inirite sequences 3. (U) Abily to use built-in functions ike sun that work with terables 4. (U) Possibilty of saving memory (0.8. range )Doc ID: 6589693 (U) Generators (U) Generators are iterators with a much lighter syntax. Very simple generators look ust ke list comprehensions, except they're surrounded with parentheses () instead of square brackets {] . More complicated generators are defined lke functions, withthe one difference being that they Use the yield keyword instead ofthe return Keyword. A generator maintains state in between times when itis called; execution resumes stating immediately ater the yield statement and continues untl the next yield is encountered. y= (xx For x So range(0)) print(y) # hen ef xsquared(): for 4 in range(se): yield $1 ef xsquared_int(): wile True: yleld xx squares = [x for x dn xsquared()) print(squares) {(U) Another example..days of te week! ef day_of week() 1-0 days ~ ["Ronday", "Tuesday", "Wednesday" "Thursday", "Friday", "Saturday", "Sunday"] while True: yield days[127) fed ay oF weok() Anmport randon f4eFsnosday(prob-.01): r= random. randoe() 46 6 © prob: return “snowday!™ se: ‘return “regular day."Do ID: 6559693 for x An day_of week(): ‘today ~ snonday() print(x +" 4s a” + today) 4 today = “snowday!” ‘break weekday = (day for day in day_of week()) ext (weekeay) (U) Pipelining {(U) One powerful use of generators isto connect them together nto a pipeline, where each generator is used by the next. Since Python evaluates (onerators "lazily" Le. 26 needed, this can increase the speed and potential alow steps to run concurrenly. This is especialy useful fone or two stops can take a longtime (e.9. database query). Without generators, the long-unning steps will become a bottleneck for execution, but ‘generators alow other steps to proceed while waling fr the long-unning steps to finish. Amport random # Get the fractional port of @ string representation of @ float ef Frac_part(v): v= str(v) 4, f= vsplit(.") return # 4 tradtttonat approach results = (1 for {in range(20): = randon-randoa() °18 # generate a ronfon maber ristr = str(r) 1 convert {¢ t0 9 string rofrac = frac_part(rstr) # get the fractional part Flout = float("®." + r_frac) # convert it back to @ float results. append(r_out) resultsec ID: 689693 1 generator pipetine rand_gen = ( randon.randon() * 208 for 4 in range(20) ) ste_gen = ( str(r) for r in rand_gen ) frac_gen = ( frac_part(e) for r i str_gen ) fout.gen = ( flest("®."sr) for r in frac_gen ) results = Mist(out_gen) results (U) Sorting (U) In Python 2, anything iterable can be sorted, and Python will happily sort it for you, even if the data is of mixed types--by default, it uses the Satin tmp uncon wich almost aye doos someting (oxcpt wth complex nubs) However, he ecu maya pe what you expect (Un Pte 3, terable objets must have the _3¢__ (1 = less han) funcon expt defined inorder to be sortable (U) The builtin function sorted) returns a new list withthe data from x in Sorted order. The sort. method (for Lists only sorts alist in-place ‘and returns None snt_data = (20, 1, 5, 6 2] sorted(int data) snt_data int_data.sort() snt_data (U) To specify how the sorting takes place, both sorted and sort take an optional argument called key. key specifies a function of one ‘argument that i used to extract a comparison key rom each list element (e.g. key=str. Lower ). The default value is None (compare the elements rect). users = [‘hackert", ‘"TheBoss", “botman', ‘turingTest"] sorted(users) sorted(users, key-str. lower) (U) The __1+__ function takes two arguments: se1f and another object, normally ofthe same type,oe ID: 6559693 class. conparablecnp (complex) def _it_(self, other ‘Feturn abs(Self) < abs(other) aeay sea aco a1 = conparablecnp(a) bt = conparablecep() arch (ba, at] sorted(e) (U) Here's how it works: 4. the argument given to key, must bea function that takes a single argument; 2 intemal, sortea creates function calls key(iten) on each tem in the list and then 4. sors the orginal list by using —_t__ on the results ofthe Key(iten) function (U) Another way to do the comparison is to use ey ‘def magnitude key(a): return (a°a.conjugate()) real ragnitude.key(2043) sorted( (5025, 15, -24, 35005], keyomagnitude key) (U) In many cases, we must sort list of dictionaries, lists, or even objects. We could define our own key function or even several key functions for ifn! keting mato List_to_sort = [{"Inane":"Jones*, “fnane':'Sally"), {°Inane dones', “fnane’ "Jerry", {Iname*:"Saith', “frane’:John"}, (le return List stent Iname")oc ID: 6689693, ef Fane sorten(1ist_stem): return List_stem{ fname”) ef inane_then_frane_sorter(List_iten) return (1ist_iten{"Inane'], list_ite['fane']) sorted(List_to_sort, key-lnane_sorter) sorted(List_to_sort, key-frane_sorter) sorted(ist_to_s rt, key-lnane_then_frane_sorter) (U) While i's good to know how this works, this pattem common enough that there isa method inthe standard library operator package o doit ‘even more concisely, {import operator Inane_sorter = operator. stengetter(“Inane") # sone of previous nose sorter (U) The application ofthe Ltengetter: method retums a function thats eqivalent tothe nane_sorter function above. Even better, when passed ‘muliple arguments, it retums a tuple containing those tems inthe given order. Moreover, we dont even need to give ita name fst, ine to do this sorted(List_to_sort, key-operator. itengetter("Inane")) sorted(List_to sort, key-operator.itengetter("Iname", "frame")) # same as using Lnome_then fnane_sorter (U) Touse operator.stengetter with List oF tuple 8, give it integer indices as arguments. The equivalent function for objects Is operator attrgetter (U) Since we know so much about Python now, its not hard to figure out how simple operstor.stemgetter actually is; the following function is essentially equivalent: ef itongetter_clone( args): et F(iten) return tuple(iten{x] for x in args) return # (U) Obviously, operator.stengetter and itengetter_clone are not actually simples just that most ofthe complet is hidden inside the Python intemals and arses out ofthe fundamental data mod (U) Duck TypingDoe ID: 6503 {GVA ne magic methods we've discussed are examples ofthe fundamental Python principle of duck typing: "fit walks lke a duck and quacks lke a duck, it must be a duck" Even though Python has isinseance and type methods, i's considered poor form to use them to validate input Inside a function or method. I verfcation needs to take place, it should be restricted to verifying required behavior using hasattr. The benefit of this approach can be seen inthe builtin sun function help(sun) (U) Any sequence of numbers regardless of whether i's a List , tuple, set , generator, or custom erable, can be passed to sum (U) The folowing is @ comparison of bed and good examples of how to write @ product function: ef List_prod(to_multiply): AF Asinstance(to multiply, List): # don't do this! ‘accumulator = 1 for 4 in to.multiply: accumulator *- § return accumulator else: raise Typetrron("Argunent to sultiply must be list") ef generic_prod(to_multiply Af hasattr(to_multiply, "_tter_') or hasattr(to_aultiply, "_getiten__ ‘accumulator = © for 4 in to_multiply: accumulator "= § ‘return accumulator aise Typetrror("Argunent to_sultiply must be a sequence") List_prod({3,2,2)) List_pred((2,2,3)) sgeneric_pred((22,3)) (U) Having given that example, testing for iterabilly is one ofa few special cases where isinstance might be the right function to use, but notin ‘the obvious way. The collections package provides abstract base classes which have the express purpose of helping to determine when an ‘object implements a common interface. (U) Final, effective use of duck typing goes hand in hand with robust error handling, based on the principle that “its easier to ak fo forgiveness than permission."DoeDe69603 Exercises 1. Ata meta to your RangedQuey class to alow instances fhe cass tobe sorte by ‘start dt! 2 vito an trata aos Revereoter at thes atl ad erates fom he revere recon 3. Wite a generator which wil iterate over every day in a year. For example, the ist output would be ‘Monday, January 1. ‘4 Mody the generator from exercise 2 50 the user can specify the year and ital day ofthe week UNCLASSIFIEDDo ID: 6589693 Pipelining with Generators Created over 3 years gag by[ TT” L:Pytnons thumbnail (U) Defining processing pipelines with generators in Python. I's simply awesome. Pipelining with Generators Imagine you'e doing your laundry. Think about the stages involved. Roughly speaking, the stages are sorting, washing, drying, and folding. The ‘beauty though is that eventhough these stages are sequential, they can be performed in paral. Ths is called pipelining, Python generators make pipelining easy and can even dary your code quite a bit. By breaking your processing into distinct stages, the Python Interpreter can make bettor use of your computer's resources, and even break the stages out into separate threads behind the scenes. Memory is. also conserved because values are automatically generated as needed, and discarded as soon as possible. {A prime example of his is processing results from a database query. Oten, before we can use the results ofa database query, we need to clean thom up by running them through a series of changes or ransformations. Pipelined generators are peroct fr this. ‘fron pprint import pprint import randon A Silly Example Here we're going to take 200 randomly generated numbers and extract ther fractional parts (the part after the decimal point). There are probably ‘more efcient ways to do ths, but we're doing to do itby sling out the string ito two pars, Here we have a function that simply returns the integer part and the fractional pat of an input oat as two strings in a tupleDoe ID: 6589693 ef split_Aloat(v): ‘Takes a Float or string of 2 Flost land returns a tuple containing the Anteger part and the fractional part ff the number, as strings, respectively. v= str(v) 4, f= vesplitc.") return (1, °8."+6) The Pipeline Here we have a pipeline of four generators, each feeding the one below it. We pprint ou the final resulting star all the stages have comple See the comments after each ine for further explanation, rand_gen = ( randon.randon() * 160 for i in range(200) ) # generate 200 random floats between @ and 108, one at o tine results = ( split float(r) for rin randgen ) # call our split floot() function which will generate the correspoding tuples results - ( r{1] for r An results ) # we only care about the fractional part, s0 only heep that part of the tuple results = ( float(r) for r in results ) # convert our fractional value fron @ string back nto o float porint(List(results)) # print the Final results Why not a for-loop? We could have put al the steps of our pipeline into a single foroop, but we get a couple advantages by breaking the stages out into separate ‘onerators: + There's some clay gained by having distinct stages specied as a pipeline. People reading the code can clearly see the transforms. + Ina for-Joop, Python simply computes the values sequential; there's no chance for automatic optimization or mult-hreading. By breaking ‘the stages out, each stage can execute in paral, just Ike your washer and dyer. Another (Pseudo-)Example Hore’ a pseudo-example querying a database that returns JSON that we need to conver to lists.Bo ID: 6589693 mort json results = ( json-loads(result) for result in b_cursor.execute(my_query) ) results = ( r{'results'} for r in results ) results = ( [ r{'nane'], r{'type'], r['count'], e['source'] ] for v in results ) Filters \We can even fitr our data in our generator pipeline. results = (r for rin results if r[2] > @) remove results with a count of zere foo(results) # do something else with your resultsoe ID: 6589693 Lesson 11: String Formatting Upisied gmat ao [Th st a (U) Lesson 11: String Formatting (©) G)-P-L, 86-36 Fy) 2: UNCLASSIFIED (U) Intro to String Formatting (U) Sting formating isa very powerful way to display information to your users and yourself. We have used it through many of our examples, such as ts: ‘This As a formatted String ()° format(*---oht I'm a formatted String argumenté- (U) This is probably the easiest example to demonstrate. The empty curly brackets {) take the argument passed into. format (U) Here's a more complicated example: (2) {4} and (0) format (‘Heney", 1", “Bob (U) Arguments can be positional, as illustrated above, or named like the example below. Order does matter, but names can help. {ho} 45 really {what}!" format (who-"Tony’, what="awesone") {(U) You can also format ist:oc ID: 6689653 cities = ["Dallas', ‘Baltimore’, ‘0C", ‘Austin’, “New York") {o[41) 4s a really big city.” format(cities) (U) And dictionaries: over to_upper = {" “This 45 a big letter {0[a])*format(lower_to_upper) # notice no quotes around o "This 45 a big letter {Lookup[a]}*.format(Lookup-lower_to_upper) # can be noned for 1stele, big in lower_to_upper.stens(): print([--0{@:10} -- (1:10}<--]" format(little, big)) {(U) Ifyou actually want to include curly brackets in your printed statement, use double brackets lke this: (( }) “({0)) (0)" format(“imere do I get printed?*) {(U) You can also store the format sting ina variable ahoad of ime and use it lator: ‘the way_1_want_it = "{@:>6) = {@:>#16b) = (0:406x)" for 4 in 1, 25, 458, 7890: rint(the say {/want_st-format(4)) (U) Format Field Names (U) Here are some examples of field names you can use in cuty brackets within a format string. {fled name>) + (U)1: the second positional argument + (U) name : keyword argument + (U) Oar: attribute named var of the frst positional argument + (U) 310]: element 0 ofthe fourth positional argument + (U) me_¢atalkey]: element associated withthe specic key string key’ of me_data (U) Format Specificationoo ID: 6680693 (G) Won using a format specification, it flows the feld name within the cuty brackets, and its elements must be in a certain order. This is only for reference; fora full description, see the Python documentation on sing ormatig. {efield names :cformat spee>) 1. (U) Padding and Alignment > align right «< :align let += only for numeric types +» seanter 4.(U) Sign ‘+= pref negative numbers with a minus sign + + slke ~ but aiso pref postive numbers with a + +" lke = but also prefix positive numbers with a space 1. (U) Base Indicator (precede with a hash & lke above) + 0» = binary + 00 : octal + 0x :hexadecimal 4. (U) Digt Separator +, use a comma to separate thousands 1.(U) Fels wien + leading © : pad with zer09s atthe front 1. (U) Fila Type (eter ong which type of value should be formatted) string (the detaut) binary ‘decimal: base 10, octal hex uses lower case laters hex uses upper case lke < , use locale settings to determine decimal point and thousands seperator + no code integer: tke + = exponential with salle + s exponential wit big E + fixed point, nan for nota number and nf for infinity F same as F but uppercase aN and NFee rt + 6 slike « but uppercase +n cale settings tke + times 100, displays as F with a x ‘+ no code decimal: ike «precision of twelve and always one spot ater decimal point 4.) Vaile with (U) New in Python 3.6: f-strings 4 had 'f* before the string to create an f-string 1 expression added dtrectly inside the “(}° brackets rather than after the format stotenent Fra = OO ry_rane = ‘Bob! My name As {my_name}* (U) Examples “40:(1)2)0)" foent(o076.5832, 18, 3) “ *¢9:4010.4#)° format(-123.456) 20.48)" Format(-123.456) for 4 An range(2, 6): print("{0:10.(1}4)" format(123.456, 1)) “value :276.542, ‘width':15, ‘preciston':5) (0{ value] :(0[width]}.{0[preciston}}}*.format(v) data = [('steve', 59, 202), ('Smantha’, 49, 156), ("Dave", 61, 135)] ‘for nane, print weight in dati 12s) (1:44) (2:44)" format(nane, age, weight))oc ID: 6589693 # some as above but with f-strings data = [('Steve', 59, 202), ("Samantha’, 49, 156), Dave", 61, 335)] for nane, age, weight in data: rine (#°(nane:<12s) {3 {weight:ad)") UNCLASSIFIEDoe ID: 6589693 COMP3321 Day01 Homework - GroceryList Upate aa S er ciara pie (U) Homework for Day0t of COMP3321. Task is to sort items into bins. (U) COMP3321 Day01 Homework GroceryListoo ID: 6689653 nyorocerylist jpples", “bananas”, “ailk", “eggs”, “bread”, “Shanburgers", "hotdogs", "ketchup", “grapes”, “tilapia”, “sweet potatoes", “cereal”, “paper plates”, “napkins”, “cookies”, “Hee creas”, “cherries”, “shampoo") 18 Teens by category vegetables = ["sweet potatoes", "carrots", "broccoli", "spinach", “onions”, "mushroons", "peppers] fruit = ["bananas", "apples", "grapes", "plunbs", “cherries”, “pineapple"] cold itens = ["eges", "milk", “orange juice", “cheese, “ice crean"] proteins = ["turkey", "tilapia", "hamburgers", "hotdogs", "pork chops", "han", “neatballs"] boxed items = ["pasta", “cereal”, “oatmeal”, "cookies", "ketchup", “bread”] paper_products = ["totlet paper, “paper plates", "napkins", "paper towels] ‘tolletry_itens ~ ["toothbrush", “toothpaste”, “deodorant, "shampoo", "soap" #8 Wy items by category ny_vegetables = [] my fruit = y_cold_ttens = () myproteins = (] sy toxed items sy_paper_products. ry tolletry items a a (U)Filin your code below. Sort the items in myGroceryList by ype ino appropriate my category lists using looping and decision making print("My vegetable List: ", my_vegetables) print (My fruit List: *, ay fruit) print("My cold Sten List: ", my_cold_ items) Dprint("My protein list: “, ay_proteins) print (My boxed iten List: ", my boxed ttens) print ("My paper product List: *, my paper_products) print ("My tofletry item list: *, mtoiletry_itens)oc ID: 6689604 Dictionary and File Exercises Updated over 2 years ago by in ci’ Bees OOO CUO (U) Dictionary and file exercises for COMP3321. Lists and Dictionary Exercises Exercise 1 (Euler's multiples of 3 and 5 problem) we ist all the natural numbers below 10 that are multiples of 3 or 5, we get 3 6, 6 and 8. The sum of these multiples is 2. Find the sum ofall the muliples of 3 or 5 below 1000. multiples 3 = (1 for £ in range(3,1020,3)] multiples 5 = [1 for 1 in range(5,1020,5)] multiples = set((multiples 3 + multiples_)) # set will renove duplicate numbers sum(multiples) # odd oll the numbers together you can also do this in one line: sum([1 for 4 in range(3,1000) 4 1 X 3 == © or 4x 5 == 0]) Exercise 2 Write @ function that takes a ist as a parameter and retums a second list composed of any objects that appear more than once inthe orignal ist aeproved for Relel FOIA Case 1081654.5.6) should retum (3,6) + what should duplicates((cow’pi'goat’ "horse pig) return? # you can use a dictionary to Keep track of the number of tines seen ef duplicates(x): ‘up(i} = dup. get(aye)s result = (] for i in dup-keys(): A dupl) > 1 result. append(1) return result X= 11,24356470304,545] Gupticates() ‘you con also just use Lists. def auplicates2(x) on = 0) for An x: Af x-count(4) » 2 and 4 not Sn dup: ‘dup. append(i) return dup ¥ = [cow "pig" "goat, "horse dupticates2(y) 2 = ['2016", "2615", "2016"] ‘dupticates(2) Exercise 3 \Wote a function that takes a potion mark as input and returns the ful classification + convert_classiiation(U/FOUO) should rtum 'UNCLASSIFIEDI/FOR OFICIAL USE ONLY’ + conver_classiication(S/IREL TO USA, FVEY) should return ‘SECRETI/REL TO USA, FVEY"Dec ID: 689604 1 just create 0 “Lookup table” for potentol portion marks full_classifications = {'U//FOUD":UNCLASSIFIED//FOR OFFICIAL USE ONLY", 'C//REL TO USA, FVEY':'CONFIDENTIAL/ REL TO USA, FVEY" '5//MEL TO USA, FVEY':SECRET//REL TO USA, FVEV', S//ST//REL TO USA, FVEY': 'SECRET//SI//REL TO USA, FVEY" "TS//REL. TO USA, FVEY": "TOP SECRET//REL TO USA, FVEY", "TS//S1//REL TO USA, FVEY": "TOP SECRET/SI//REL TO USA, FVEY"} def convert_classification(x) return full_classifications.get(x, UMKNOMN") # Look up the value for the portion mark convert_classtfication(*U//F00') convert_classification(S//REL TO USA, FVEY") comvert_classification(¢//S=") File Input/Output Exercises “These exercises build on concepts in Lesson 3 (Flow Control, 9. for loops) and Lesson 4 (Container Data Type, 9, dictionaries). You wil use all these concepts together with reading and wating from fles First, Get the Data ‘Copy the sonnet from hiins://urn.nsa.ic.gov/t/ix6am and paste it into a new text file named sonnet.txt. Exercise 1 rite function called file_capitalize( that takes an input fle name and an output fle name, then writes each word from the input fle with only the first letter capitalized to the output fle, Remove all punctuation except apostrophe, File capitalize("somet.tet", “sonnet_caps.txt") => capitalized words written to sonnet_caps.txtoc ID: 6589654 # use help("") to see what each of these string methods are doing ef capitalize( sentence): words = sentence.split() # use split to split the string by spaces (1-e., words) new_words = [ word.strip().capitalize() fer word in words } # coptiaLize each word return * *Join(new words) # create and return one string by combing words with ‘def renove_punct (sentence): 4 since replace) method returns @ new string, you can choin calls to the replace() 4 ethod in order to renove alt punctuation in one Line of code return sentence.replace(".’, "').replace(',", '*).replace(':', '*).replace(';", ") def Fle copitalize(infilenane, outflle nae): Anfile = open(infile.nane, 'r*) # open the input File outfile = open(outfiienane, ‘w") # open the outpu Fite for Line in infile: # loop through each Line of input outFile weite(capitalize(renove punct(Iine)) + *\n') # unite the copttalized version to the output file Anfile.close() # finally, close the files ovtfile.close() FAle_capitalize("sonnet.txt", ‘somet_caps.txt!) Exercise 2 “Make function called fle_word_count() that takes a file name and returns a dictionary containing the counts for each word. Remove all punctuation except apostrophe. Lowercase all words. File_word_count("sonnet.txt") => ("it's 4, ‘neec ID: 6689654 ef #L1e word count (inFile_nane): sword_counts = (} with openinfilenane, ‘r') as infile: # using ‘with’ so we don't have to close the File ‘for Line in infile: # loop over each Line in the file words = remove punct(Line) # se can use the renove.punct fron exercise above ords = words.split() # =pLit the Line into words for word in words: # Loop over each word ord = word. strip()-Lower() "add one to the current count for the word (start at @ {f not there) word_counts{word) ~ word.counts.get(word, ©) + 3 ‘retuen word.counts # return the whole dictionary of word counts counts ~ Fle word_count( sonnet. txt") Extra Credit Write the counts dictionary oa fle, one key-value per in. ‘def write_counts(outfile name, counts} with open(outfilenane, "w', encoding-"utf-8") as outfile: 4 to Loop over a dictionary, use the ttens() method 1 {tens() will return 2-eLement tuple contetning @ hey and a value 4 below we pull out the values from the tuple into their own vartables, word and count for word, count An counts. itens(): outfile.write(vord + ':* + str(count) + "\nt) # write out in he value format wrlte_counts(‘sonnet_counts.txt", counts) # use the counts dictionary fron Exercise 2 aboveoc ID; 6649694 Structured Data and Dates Exercise ptt ge 9 year 200 fT momma OoOoOc (U) COMP3321 exercise for working with structured data and dates. Structured Data and Dates Exercise ‘Save the Apple stock data from hiios/um.nsa c-9o¥/VO9r to aapLsy Use DictReader to read the records. Take the daly stock data and compute the average adjusted close (Adj Close”) per week. Hint: Use Isocalencar() for your datetime object to get the week number. For each week, print he year, month, and average adjusted cose to two decimal places. Year 2015, beck 23, Average Close 107.48 Year 2015, Week 22, Average Close 105.10oc ID: 6559694 ‘fron csv import DictReader fron datetine import datetine ‘def average(nunbers): Af Len(nunbers return 9.0 return sun(runbers) / float len(rumbers)) ef get_year_veok(record): ‘dt = datetine.strptine(record[ Oate"], "2¥-tm-xd') return (at.year, at. ts0calendar()(1]) def get_averages(data): aves = 0 for year_week, closes in data. ttens(): ‘avgs[year_week] = average(closes) return aves ‘det weokly_surmary reader): weekly data = () for record in reader: year_weck = get_year_veok(record) Af year_week not in wookly. data: weekly_dataLyear_veet] = [] weekly_data[year_week] append(Float(record Adj Close"})) ‘return get_averages (weekly data) ef F.Le_weekly_surmary(infile_name): with open(infile mane, ‘r') as infile: return weekly, summary OictReader(infile)) def print_weekly_summary (weekly_dats for year_week in reversed(sorted(weekly_data.keys()))oc ID: 689604 year = year woek(0) week = year_week{ 1] avg = weekly datalyear_week) Drint(Year {year}, Week (week), Average Close {avg:.2f}' format(year-year, week-veek, avg-2v@)) ata = File woekdy_summary("aplcs¥") print weekly_suenary(data) Extra Use csv DictWriter to write this week data out toa new CSV fle ‘from csv Anport Dictwriter def write weokly.sunmary(weekly data, outfile_nane) headers = [ "Year", ‘week’, "Avg" ] ith open(outfile_nane, 'w', newline-"') as outfite: weiter = Dictweiter(outfile, headers ) weiter wrsteheader() a-keys())): vyear_week(], for year_week in reversed(sorted(weekl) rec = { "Year": year_week[o], “week” writer weiterow(rec) weekly datalyear_week] } data = File weekly sumary(‘aapl.csv') write weekly, summary (data, ‘aapl_sumary.csv") Extra Extra Use son dumps( to write a JSON entry for each week on a new line. Amport json def urite_json_weekly_summary (weekly data, outfile nane): with open(outfilenane, 'w") as outfile: ‘for year_week in reversed(sorted(weekly data. keys())): ree = { ‘year’: year_week{0}, "week': year_week[ i], ‘ave loutfie.write(json.dumps(rec) » "\n")oc ID: 6559604 data ~ File wookdy_sumary(aapl.csv") write_json weekly_sumary(data, ‘29p1.Json’)Dec ID: 6559604 Datetime Exercise Solutions Getta eee i ii 32 * pee se ovo (U) Solutions forthe Datetime exercises (U) Datetime Exercises (U) How tong before Christmas? Anport datetine, tine print(datetine. date(2017, 12, 25) - datetine.date.todey()) (W) Or if youre counting the microseconds: print (datetine.datetine(2017, 12, 25) - datetine.datetine.today()) (U) How many seconds since you were born? birthdate = datetine, datetine( 1985, 3, 23) ‘tine since birth ~ datetine. datetine.today() - birthdate print("(:,)" format (tine_since birth. total_seconds())) (U) What is the average numberof days between Easter and Christmas forthe years 2000 - 29887oc ID: 6689694 fron dateutil.easter taport easter total = & span = range(2000, 3020) for year in span: otal + (datetine.date(year, 12, 25) ~ easter(year)) days ‘average = total / len(span) print ("(:6.4°)' Format (average)) (U) What day ofthe week does Christmas fll on this yoar? datetine.date(2015, 12, 25).steftime("XA") (U) You get a intercepted emai wth a POSIX timestamp of 1495074325. The ema s from the leader of a Zendian extremist group and says that ‘there will be an attack on the Zendian capitol in 14 hours. In Zendian focal time, when wil the attack occur? (Assume Zendia sin the same time ‘one as Kabul) Amport pytz lute tz = pytz.timezone("Ete/UTC') ‘onail_tine_ute = datetine.datetine.fromtinestarp(1635078525, tz-ute_t2) attack tine_ute ~ enail_tine utc + datetine.tinedelta(hours@14) Zzendia_tz = pytz.tinezone(Asia/Kabul') attack tine_zendia = attack tine_ute.astinezone(zendia tz) print enati_tine_ute) print (attack tine ute) print (attack tine_zendia)oe ID; 6589694 Object Oriented Programming and Exceptions Exercise grees a yeas af Ti Se ate (U) COMP3321 exercise for object oriented programming and exceptions. Object Oriented Programming and Exceptions Exercise Mak a ass cle Symbol at hots data fora sock synbo wih th flowing proper: self.nane self.daily_data It should leo have the following functions: def _init_(self, nave, input_file) def data for date(self, éate_str) Init(sef, name, input fle) should open the input fle and read it with DictReader, puting each entry in self. daily_data, using the date strings as the keys. Make sure to open the dally data fie within a trylexcept block in case the fle does not exist the fle does not exist, sot sof daily data to an cempy dictionary. ta_for_dato(sol,date_str) should take a date sting and retum the dctionary containing that days data. If there is no entry for that date, return an ‘empiy dietonaryoe ID: 689694 Tests “Make sure the folowing execute as speciid in each comment. You can get the aaplcsv fle from hios:/un.nsa.i.cov)uOari. The apple.csv fle should not exis. 51 = Symbol("ARPL*, ‘aaplesv") print(s1.data_for_date("2615-08-10")) print(st-data_for_date("2615-08-09")) 52 = symbol("AAPL', ‘apple.csv") print(s2.data_for_date("26i5-08-10")) Drint(s2.data_for_date(*2635-08-09")) 4 should return a dictionary for that date 4 should return an eapty dictionary 4 should not raise an exception! 4 should return an eapty dictionary 4 should return an empty dictionaryDoe 1D: 6589695 Module: Collections and Itertools wr ated amost 2 yeers ago by[ Jin Coie 32s oc (U) Module: Collections and ttertools (U) Any programming language has to strike a balance between the number of basic elements it exposes, ke control structures, datatypes, and so ‘orth, and the utity of each one. For example, Python could do without tuple s entrely, and could replace the ict witha list of 1ist sor even ‘single st where even-numbered indices contain keys and odd-numbered indices contain valves. Often, there are situations that happen so ‘commonly that they warrant inclusion, bu inusion inthe builtin library isnot quite justified. Such is the case with ‘the collections and itrtools modules. Many programs could be simplified wih a. defauitsict , and having one availabe witha single fron collection import defaultaict fs much better than reinventing the wheel everytime its needed. (U) Value Added Containers with collections (U) Suppose we want to buld an index for a poem, so that we can ook up the lines where each word occurs. To do this, we plan to construct 2 ")) 4 -update(na John", action="ran")oe ID: 6559698 “{@{nane]} {@{action}} to (O{obsect]}" Format) 4 # “object” added to 4 (UA counter I ike a defoultsict(int) with additional features If given a 2st or other iterable when constructed, it will create counts of ll the unique elements it ses. It can also be constructed from a dictionary with numeric values. Ithas a custom implementation of update and some specialized methods, ike most _conmon and subtract ‘fron collections import Counter vord_counts = counter(poem.split()) word_counts.most_common(2) ord_counts.update("Ianb Lanb lanb stew" .split()) word_counts.ost_comon(3) € = counter(ar3, bei) 4 » counter(art, b-2) cod ed 1 Dia you get the output you expected? (Oe cad ela (U) An ordereabict is a dictionary that remembers the order in which keys were originally inserted, which determines the order fr its iteration, ‘Aside fom that, thas a popitem method that can pop from ether the beginning or end ofthe ordering. (U) nanectople Is used to create lightweight objects that are somewhat ike tuples, n that they ae immutable and atributes can be accessed with [) notation. As the name indicates, attributes are named, and can also be accessed with the . notation. tis most often used as an ‘optimization, when speed or memory requirements dictate thal a cic or custom object isn't good enough. Construction of a nanectuple is Somewhat indirect, as nanestuple takes field specications as strings and returns @ type , which s then used to create the named tuples. named tuples can also enhance code readabilty. ‘from collections import nanedtupleoc ID: 6689685 Person = nanedtuple("Person", bob = Person(nane="Bob print ( "Xs 1s a 2d year-old Xs" x bob) # 2.x style string formatting rint( "{) 1s a {) year 0" Format(“bob) ) Print( ‘25 48 9 2d year-old Xs" X (bob.name, bob.2g6, bob. gender) ) print( "{) As 2 Q year-old {)'.Fformat(bob.nane, bob.age, bob. gender) ) veo} boot mane") 1 Typetrror bob nane PrAnt( "R(nane)s As a X(age)d year-old X(gender)s” X bob) # Doesn't work print( "{name) is a (age) year-old {gender} format(*bob) )# Doesn't work print mane) 1s 2 {0.age) year-old {8.gender}'.format(bob) ) # Works! (U)Finaly, deqve provides queue operations. rom collections import deque = deque("ghi") __# make 0 new deque with three ‘tens .append('3") 4 add a new entry t0 the right side A.appendlefe('#) ——# odd 0 new entry to the Left side 4.poplert() ‘+ return ond remove the Leftmost tem a.rotate(s) 4 ight rotation Gentendlefe(‘abe") —# extendLeft() reverses the input order (U) The collections module also provides Abstract Base classes for common Python interfaces. Their purpose and use is curently beyond the ‘cope ofthis course, but the documentation is reasonably goodoc ID: 6589588 (U) Slicing and Dicing with itertools Given one or more List, iterators, or other itrable objects, there are many ways to sce and dice the constituent elements. ‘The itertools module tries to expose building block methods to make this easy, but also tes to make sure that ts methods are useful situations, so the documentation contains @ cooktonk of common use cases. We only have time to cover a small subset of ‘the itertoots functionality. Methods from itertools usually return an iterator, which is great for use in loops and lst comprehensions, but not so good {or inspection; inthe cade blocks that folow, we often call List on these things to unwrap them, (U) The chain method combines iterabes into one supersterable. The groupby method separates one iterator into groups of adjacent objects, possibly as determined by an optional argument-his can be tricky, especially because there's no look back to see Ifa new key has been ‘encountered previously Amport itertools 2 variety of Lst(stertoots. ehain(range(s)+{545])) => [8114243444545] size_groups = itertools. groupby((1,1,2,2,2,'B"s"B,3,4,2,3,21) [ekey, List(vals)) for key, vals in size groups] (U) A deeply nested for loop or lst comprehension might be beter served by some ofthe combinatoric generators ike product , permutations, (or combinations ter_product = itertools.produet([1,2,3],1'2", en List(iter_product) Ater_conbi = itertools. combinations ("abet List combi = List(iter_ combi) Aist_conbi ‘ter_permutations = itertools.permutations("abcd”,3) List(iter_permutations) (U) stertoots can also be used to create generators: counter = Stertools.count(®, 5) next (counter) print(list(next(counter) for c in range(s)))ose oe (U) Be careful... What's going on here?!? counter = Stertools.count(0.2,0.1) for ¢ in counter: print (c) Ae > 15: break cycle = itertools.cycle(*ABCDE") for 4 in range(10): print (next (cyele)) repeat = itertools. repeat ‘again! ") for 4 in range(s): rine (next (repeat) ) repeat = itertools.repeat(*again!", 3) for i in range(s): Drint (next (repeat) ums = range(19,0,-1) ay.2ip = zip(nuns, itertools.repeat(‘p')) ‘for thing in ay_zip: print (thing)Doc ID; 6589695 Functional Programming we Created over 3 years ago by[ Jin GoM 325, cad Sse : Ommome (U#FOUO) A short adaptation of FA practical introduction to funetional programming® in Python to supplement COMP 3321 material Aso Gacusses lambda. UNCLASSIFIED (U) Introduction (U) Ata basic evel, there are two fundamental programming styles or paradigms: + imperative or procedural programing and + declarative or functional programming, (U) Imperative programming focuses on teling a computer how to change a program's stalo-its stored infrmation-step by slep. Most programmers start out learning and using tis style. I's @ natural outgrowth ofthe way the computer actualy works. These instructions can be ‘organized ino functions/procedures (racedural programming) and objects (object-orientedprogramming), bu those stylistic improvements remain imperative at heart. (U) Deciaratve programming, on the other hand, focuses on expressing what the program should do, not necessarily how it should be one. Functional programming isthe most common favor ofthat. I treats a program as if itis made up of mathomaticastye functions: fora given input x, running trough function £ wil aways give you the same output f(x) , and x itself will remain unchanged afterwards. (Note tha this isnot necessariy the same as a procedural-stye function, which may have access to global variables or other inputs" and which may be able to ‘modify those inputs directly.)Doe ID: 659695 (U) TL;DR (U) The key distinction between procedural and functional programming is this: a procedural function may have side effects-it may change the sate ofits inpus or something ouside ite, giving you a diferent result when runing it Second tie. Functional programming avoids side effects, ensuring that functions don't modi anything outside themselves. (U) Note (Uy Tos contra ft rtsook have bon benowed tom fe begin fener Feganmang Atl me vervan of Pat esa an bo oan ats fsa AT Ues Pon?) (U) Functional vs. Not (U) The best way to understand side effects is with an example (U) This function is nt funetiona: def sncrenent() tlobal a practical introduction to functional (U) Tis function is funciona: ef increment (a): return a+ 1 (U) Map-Reduce (U) Lets jump into functional coding. One common use is map-reduce, which you may have heard of. Let's see if we can make sense oft (U) map (U) Conceptually, map isa function that takes two arguments: another function and a collection of items. will 4. run the function on each item ofthe original collection and 2 return a new collection containing the results,Pec ID: S25 eavng th cxiginal collection unchanged (U) In Python 3, the input collection must simply be Herable (e.g. List, tuple, string). Is map function retums an iterator that runs the input Tuncton on each item of terable (U) Example 1: Name Lengths (U) Take alist of names and get alist ofthe name lengths: rane_lengths = map(len, ["Mary", "Isla", “San")) print (list(name_lengths)) (U) Example 2: Squaring (U) Square every number in a ist squares = map(lanbda x: x" x, (8) 3, 2, 3, 41) print(list(squares)) (U) A digression on lambda (U) So what's going on with that input function? anda wil let you define and use an unnamed functon. Arguments ftbetween the Isnods. and the colon while the stuf ater the colon gets impli returned (i. without explty using a return statement). (U) Lambdas are most useful when: + your function is simpe and + you only need 1 use it once. (U) Consider the usual way of defining a function: ef square(x): sauare(s) | we could have done this instead squares = map(square, [2, 2, 2, 3, 41) print(list(squares)) (U) Now let's define the same function using a lambda:‘Doc ID: 6559698 lambda x: x * x (U) Fine, but how do we call that resuitng function? Unfortunately, it's too late now; we didnt store the result 0 t's last nthe ether. (U) Lets try again ‘na_funetion variable = lambda x: x * x ‘ype(sna_function variable) ‘na_funetion_variable(s) 1 be careful! ‘ma Function variable = “something else’ 1 eur Lanbda function is gone agotn ‘ma Funetfon_variable(4) (U) Example 3: Code Names (U)Ok, back to ap. Here's a procedural way to take a ist of real names and replace them with randomly assigned code names Amport random names = ['Nary', "Isla" code_nanes = ['Ne. Pink’ —) Wr. Orange’, "Kr. Blonde" } for 4 in range(Jen(nanes)) rnanes[1] = random, choice(code_nanes) print (names) (U) Hore's the functional version: names = ['Nary', ‘Isla’, 'San'] ‘covernanes - nap(lanbéa x: random.choice({‘Nr. Pink’, “Mr. orange print (list(covernanes)) "Mr. Blonde']), nanes) (U) Exercise: Code Names...Improved?oc ID: 6659695 (6) The procedural code below generates code names using a new method. Rewrite itusing map anes = ["Rary', "Isla", ‘San] for & An range(len(nanes)): ranes[1} = hash(nanes{}) print(nanes) 1 your code here reduce (U) Reduce is the folow-on counterpart map. Given a function and a collection of items, uses the function to combine them into single value ‘and retums that result (U) The function passed to reduce has some restrictions, though. It must ake two arguments: an accumulator and an updato value. The update value is tke it was before with map ;t wil get set to each item in the collection one by one. The accumulator is new. It wil receive the output from the previous function cal, thus “accumulating” the combined value from item totem through the collection. builtin function. Python 3 moved it into the functools. package (U) Note: in Python 2, reduce was (U) Example (U) Get the sum of all items ina collection. import functools ssun = functools.reduce(lanbda a, x: 2+ x, [8 prine(sum) UNCLASSIFIED 23,40oc ID: 6689595 Recursion Examples [pore a] Spee 8s by] ote Domo oOCIes8 (U) Some simple recursion examples in Python Recursion Recursion provides a way to loop without loops. By calling ise on updated data, a recursive function can progress through a problem and traverse the options. Nth Fibonacci Number nitoswitinedia.nsaic.govlen/Fiponacci number ‘This returns the nth Fibonacel number in the Fibonacel Sequence using recursion, ef nth_Flbonacct(n): fencd return nth fibonacci(n-2) + nth Fibonacci (n-1) nth Fabonaces (20)Dee ID: 689695 Fibonacci Sequence ‘This tums alist of the frst n Fibonacci Numbers using recursion. ef Fabonacci(n, seq-(1) 4 Jen(seq)’ =n: return seq (e146 en(seq) == @: return fibonacci(n, (21) [e14€ en(seq) == 1: return fibonacci(n, (2,21) ase: rext_value = seql-2} + sea{-1] ‘return Fibonacci(n, seq + [next_value]) abonacet (5) Simple Game This simple game just takes ina ist of rine elements and tries to modfy each slot untl all the numbers from 1 to 8 are in the ist. import random‘Boe ID; 6559695 ‘def inprove(input_List, missing): ‘random index = random. choice(1ist(range(len(input_list)))) ‘random_value ~ random. choice(eissing) new list = input list{:] new list{random index) = random value return new 11st def Find missing input List): missing = [ x for x An List(range(s,10)) 4 x not Am input list ] return missing ‘def one_to_nine(input_List): print input_1ist) faissing ~ find aissing(input_list) Af Len(missing) == 0: return input_list ise: new list = inprove(input_list, missing) return one_to.nine(new List) one to_nine((2)31,3y1,441,152]) Simple Game Revised ‘This revision of the same simple game comes up with alist of possible improvements and tries to pick the best one o pursue. Youll notice that it takes fewer attempts to each an anser than the original version ofthis simple game,oc ID: 6589698 ef inprove( input List, missing): random_index = randon, choice(1ist range(Len(input_list)))) random value - randon. choice(missing) new List = input Uist{:] rnew_list{randon_index) = randon_value return new 1ist def Find_missing input_List): intssing = [ x for x in List(range(s,10)) 4 x not An input List ] return missing ‘def score(snput list): missing » find missing(input_list) return (len(eissing), input_2ist) def best_scoring list( scored lists) lowest = 100 best list = [ for x in scored lists: score = x{0] Anput_list = x{2] Af score < lowest: lowest - score best list = input list return best 1ist ef one_to_nine(input list): print input_1ist) fissing ~ find missing input_list) AF Len(missing) == 0: return input list ase: possible iaprovenents = [ improve(input List, missing) for 4 in range(len(aissing)) } scored_inprovenents = [ score(i) for 4 in possible improvements best_list = best scoring _List scored _inprovenents) ‘return one_to_nine(best_list)oe ID; 6589695 one to_nine( (1s iete3e1s3e1e301)) Simple Game as a Tree We can tink of our strategy asa tree of options that we traverse, folowing branches that show that theyre going to improve our chances of finding ‘a solution, Ths is an extremely simpified form of what many video games use for thelr A. We put our possble_improvements in a generator so they wil only be created as needed. I we put them in alist comprehension as before, then all possible improvements would be generated even though many of them wil Ikely go unused. In the end, we return any() with a generator forthe branches. Since any() only needs one item to be “True, wil retum True as So0n as a solution is found: when len(missing) ‘Youll notice that this results in more iterations of one_to_nine() than inthe previous revision. However, the previous revision also generated a lot of data that ends up getting discarded. n ather words, there's probably more processing and memory consumed by the previous revision behind the ‘def improve(input_List, missing) ‘random_index = randon.choice(Iist(range(len(input_1ist)))) ‘andom_valve ~ randon.chosca(nissing) new List = input_list{:] rnew_list{random index] = random value return neu List def Find missing(input_list): missing = [ x for x in List(range(1,10)) 4 x not in input list ] return missing ef one_to_nine(input_List, prev_missing-Non print input_1ist) fnissing ~ find mlssing(input_list) AF prev_missing 4s None: return one_to_nine(input_List, missing) e14F Len(nissing) == & return True {e146 len(rissing) > len(prev missing): return False else: possible inprovenents = ( inprove(input List, missing) for i in range(Len(wissing)) ) return any(( one_to_nine(p, missing) for p in possible improvements ))Doc ID: 6689695 one to_nine( [151 s1ae1A))Doc ID: 6559695 Module: Command Line Arguments pte gona er ago by [Jit ee = (U) Module: Command Line Arguments, UNCLASSIFIED (U) Most command line programs accept options and arguments, and many even provide help messages that indicate what options are available, ‘and how they are to be used. For example, the utiliy program mv takes two arguments, and mast often moves the fist argument (the source) to the second (the destination) thas other ways of operating, which are enabled by optional flags and arguments; for a command prompt, type mv help to see more, (U) Thore are soveral ways to enable this type of functionality n a Python program. and the best way todo it has been a source of contention In particular, this lesson wil cover the argparsemodule, which was added othe standard library in Python 2.7, and not the eptparse module which was deprecated at that time. (U) Everything passed as arguments to @ Python program is availabe in the interpreter as the ist of strings in sys. argv In an interactive Session, sys.argv always starts out as (") . When running a scrip, sys.argv(e) is the name ofthe script. We stat by examining what sys.argv looks lke. Pu the folowing commands ina fle caled argcest.py or similar: import sys print (sys. argv)oe ID; 6589695 # ...0F mohe python do it! contents = "*"import sys print(sys.argv) saith open(‘argtest.py’, °w') as f f.write(contents) (U) Close the fle and execute it from the command line with some arguments: 1 the 1" at the beplming tells jpyter to send wat follows to the comand Line Dona eratest.py “tat foo othervactar file fez > [largtest.py', ‘ox2f!, “ov, "foo", ‘--othervarsbar', “filer, “#i62"] (U) Inallof te argument parsing that follows, sys.argv will be involved, although that may happen either imply or explicitly. Athough itis often unwise to-do so within a script, sys.arev can be modified, for instance during testing within an interactive session. (U) Note thatn Jupyter you sill have. argv . butt may not be what you expect. If you look at, youl see how this Python 3 kernel i being called. Amport sys rint(sys.argv) (U) The Hard Way: getopt (U) For programs with only simple arguments, the getopt module provides functionalty similar to the getopt function in C. The main method in the module's getopt . which takes a ist of strings, usually sys.argv[:) and parses it according toa string of options, wih optional fong options, ‘hich are alowed to have more than one letter; explanations are best lf to examples. This method returns a pair of sts, one containing (option, ‘value) tuples, the other containing addtional positonal arguments. These values must then be futher processed within the program; it might be seul for instance, to put the (option, value) tuples into a dict. getopt receives an unexpected option, i throws an error. ft does not ‘receive al the arguments it requests, no error is thrown, and the missing arguments are not present in the retumed value. Amport getopt etopt.getopt('-a arg’ split(),'a:") # a expects on argument aetopt.getopt(’-a arg’ split(), ‘a:b') # no b, no problen etopt.getopt('-b ang - my-file.tet" .split(), ‘ab:') # my-file.txt ts argument, not optionee ID: 689695 etopt. getont( fang ~-outputeother-file.txt y-File.txt’ split), ‘output='1) # Long options (U) Fox programs that use. getopt , usage help must be provided manually ef usage() rine("*"usage: my_progran.py -[abh] fen, File2, . vey 4 this won't actually find anything tn Jupyter, since tpythen3 probably doesn’t have these options opts, args = getopt.getopt(sys.argv(1:], ‘abh') opt dict = dict opts) AF '305)'format("Input", *Result")) prine('=" * 70) ‘loop through the examples List to show how parseDT can successfully determine the date/tine based on both messy input 1 and messy relative tine offset inputs for ¢ in examples: de, result ~ cal.parseot(e) prine("{:<40s}(:930)" format ("™* + @ + UNCLASSIFIED » de.ctine()))Doe ID: 6589695 COMP3321 Datetime Exercises Grate apet yor apoby[ si i - cae (U) Datetime Exercises for COMP3321 (U) Datetime Exercise (U) How tong before Christmas? (U)How many seconds since you were born? (U) What isthe average numberof days between Easter and Christmas forthe years 2000 - 29907 (U) What day ofthe week does Christmas fall on this year? (U) You get a intercepted email wit a POSIX timestamp of 1495074325. The emails trom the leader of a Zendian extremist group and says that there wil be an attack onthe Zendian capitol in 14 hours. In Zendian local ime, when wil he attack occur? (Assume Zenda isin the same time zone as Kabul)oc ID: 6689695 Module: Interactive User Input with ipywidgets WO) G)-PoL, 8636 (U) Covers the ipywidgets library for getting interactive user input in Jupyter UNCLASSIFIED/HFOR OFFICIAL USE-ONEY (U) ipywidgets (U) inywidgets is used for making interactive widgets inside your jupyter notebook (U) The most basic way to get user input sto use the python builtin input functon. For more complicated types of interaction, you can use lpywicgets ‘ingut example (not using tpywidgets) asinput("Give me your input: ") print ("your input was: "+a) Amport Ipyuidgets fron ipywidgets import * Interact 1s the easiest way to get started wih ipywidgets by creating a user interface and automatically calling the specified function ef (0): return x°2 Anteract(F,x-10)ec ID: 6559695 ef g¢check,y): prine("{) ()".format(check,y)) Anteract(g,check=True,y="1 there!") ‘But, ityou need more lexibilty, you can start from scratch by picking a widget and then calling the functionality you want. Hint: you get more widget choices this way. Intsider() wentsiider() ‘You can explcy display using IPython's display module. Note what happens when you display the same widget more than oncel ‘fron rPython.display inport display display) wvalve Now we have a value from our slider we can use in code. But what other attributes or *heys" dows our slider widget have? rew_-IntSLider(max-200) ‘display(new_m) You can also close your widget weclose() ewe. elose() Here are all the avaiable widgets: Widget widget_types Categories of widgets "Numeric: int Slider, FloatSlder, ntRangeSider, FloatRangeSiier, IntProgress, FloalProgress, BoundedintText, BoundedFloatTet,IntTex, FloatText Boolean: TogoleButton, Checkbox, Validoc ID: 6539605 ‘Selection: Dropdown, RadioButtons, Select, ToggieButons, SelectMuliple ‘String Widgets: Text, Textarea ‘Other common: Button, ColorPicker, HTML, mage Dropdown (opt ons 3 reat") bt = Button(description="Click mel") display(oe) ‘Buttons dont do much on their wn, so we have to use some event handling. We can define a function withthe desired behavior and callit with the buttons on_click method. ef clicker) print "Mellow World!!!) be on elfek(elicker) ef (change): print (changet ‘new }) w= Intsiier() display(w) cobserve(,nanes="value') Wrapping Multiple Widgets in Boxes Won working with multiple input widgets, it's often nice to wrap ital ina nice litte box. ‘pywidget's provides a fow options for this-wel cover Haox (horizontal box) and vox (vertical box). HBox “This wll splay the widgets horizontallyoo ID: 6689655, Frult_Iist ~ oropdoun( options = ["apple', ‘cherry’, ‘orange’, ) Feuit_label = HOAL( value = ‘Select a fruit from the Listrtnbsp;nbsp;" ) Fruit bor = Hlox(chideen| Feult box um» “ pear] frust label, fruit_list]) VBox ‘This wl display the widgets (or boxes) vertically rnum_tabel = HTML value ~ ‘Choose the number of fruits > rum options = Intslider( rin-2, i fnum_box = Hox(childrens(num_label, nun_options)) ‘ype_label = HTML value = ‘Select the type of fruitstnbsp;anbsp;" > ‘ype_options = Radioputtons( ‘ontions-("Under-ripe", ‘Ripe’, > jtten") ‘type_box = WBox(chidren=(type_label, type_options)) ‘ruit_vbox = Wlox(children-(Frutt_box, nun box, type_box)) ‘rust_vboxDoe ID: 65g9555 Specify Layout of the Widgets/Boxes form_iten_layout ~ Layout( Aisplay="flex', flex #loe- ‘row’ Justify_content="space-between', widths "70%", align_stens~"initial’, > veggie label = HTmL( value = ‘Select a vegetable from the List:tnbsp;Bnbsp;", layout=Layout (wldth="28%", heights "65px") > vegsie_options ~ propdown( ‘options-['corn’, ‘lettuce’, “tomato', “pots layout=Layout (wldth="38X", ‘height="63px “spinach, > veggie box = Heox(chtldnen-(veggie label, veggie options), ayout-Layout(width='106x', border="solid 2px", height="100px")) vvegsie_box Retrieving Values from a Boxoc ID: 6689695 box values = 0 4 the eLenents {n a box can be accessed using the children attribute or index, box in enunerate(fruit_vbox. children): ‘for chtld in box. children: AF type(child) != ipyuidgets ul AF index = Drint(“The selected fruit is: “, child. value) box_values["fruit"] = child. value Lif index == 3: Drint("The select number of fruits is: ~, str(child.value)) boxvalues[ count") = child.value eit index == 2: Drine(“The selected type of fruit 4s: bbox_values["type"] = child.value box values UNCLASSIFIEDHFOR-OFFIGIAL USE-ONEY ts eidget_steding. HTML: ste(ehtld.value))oc ID: 6689695 pata gost2 years ago {Ji che’ es (U) Module: GUI Basics with Thiner UNCLASSIFIEDHFOR-OFFIGIAL-USE-ONLY (U) Tkinter (U) Tartar comes a pat of Pyiton so resty avait for uefa Nate: While Tetris ost ays avalbie, ytton canbe naa wet. 9g Tee nt $value wien Pyeen i cmp This a wdpt iar hat was agra dsghed Tre Te Sop lanquge, bul haw has ben pte to Pe, by, Pron C++ and mre (U)NOTE:InPyton 2 must be used ae einer wh cpt (U) Setup (vsrotoy Tis sso ann curey ern fom upton LABBENC de fo slayer tans shoud pk wang Aas py $n be cpled oe and rn sr rom MACHINESHOP yur pay reper cong (UAFOVO} Or : ito ject bck om MACHINES, owl ees ne slowng on you MAS tan ve nacassary eration (U) What's a GUI (Graphical User Interface)?Doe PGF uso mom, some ous ove hem and hate thom. Do we cari 2- of 3 denansanal 25 dmensienal/? Les loka some very base exams (U) Example 1 Iori Tt cetnort Wout eatad our fest gl! uti os do a we ye. Thats cause wo ny costed a arWomety widow that swath ecu creation...) ithe tp eel wow tat we ict for evry gua we make Pans hone wget = Arange wind == Ad ncn (U) Example 2 Aistock at wigs Anoorttkinter as th foes eH) aed = ee Labe{ean, tae-"t am 2 abe wget") screate ane Sotto acters, textos eben) sane atin rine pe) sas toe toot wri) aoe ton tot coninat) (U) Widget Types Type Description ten Use eck on butons to tgger some acon Buti leks cn be ral io actions eke by your program, autton usualy pay text but an show ropes conve rte on which you can raw graphs andor plots and alo seas the bas of your ow widgets. CheckButton A special ype of sutton that has two sales; clicking changes the stale ofthe button from one othe other. entry Used to enter single nes of txt and al kinds of input Frame A container fo other widgets. One can set the border and background color and pace other widgets init et Used to splay pieces of txt or mages, usualy ones that wont change during the execution ofthe appication.oo ID; 6689695 Type Toplevel Description Used to play a set of choices. The user can solct a single item or multiple toms fom the Ist. The Listbox canbe also rendered asa set of radio buttons or checkboxes ‘Similar Text butcan automaticaly wrap texto a particular with and height Used to puta menu in your window i you need tI comesponds tote manu bar atthe top but can also be used asa pop-up. ‘Ads choices to your Meru. Represents one of ast of mutually exclusive choices. Selecting one Rasobutton from a set desolects any others. Lets the user set numeric values by dragging a slider, Implements srling on a larger widget suchas @ Canvas. Listbox, OF Text [A muti-tine formate text widget that alows the textual content tobe "rch" l may also contin embeded images and Frane 8. ‘A special kind of frase that intoracts recy wih the window manager. Toplevl s will usualy have aie bar and features to interact with the ‘window manager. The windows you se on Your screen are most Topievel. windows, and your appcalion ean create ‘eaitonal Topleve windows tls st to do that ‘One won Cette, Lbeirane, Panelindn tea Cast, Span, Inge Cla (U) Example 3 Latsook at some oer wage xaos. ‘io rao at wats Rave or om spol wget varale “stead of using Bult python pes He mpl eer own ject or eng snarl eration The wit ‘tales Seringor, ier Daalevar and esa‘Doc ID: 6689698 apart cetera th root =H.) Au. taba(oot, testator your Parmer") -pckd) feauten(ant, tere "Seareh) peck v= te sneer) ecrecaotten(ront, texttennber Me, varabtew)pack() eereyrant at) a= teamvar() etaseostton(rent, texte", vrkaleny2, valves, puck) etocanstten(rect,terenrenale"ywarable-2y value’) pak) eoptiemena(root, vary SAAC Comey, “UHC, "Tatar, “otners") pack) { Seelar(rost, orden. "vetiea) pct) root mtaoop() (U) Three ways to configure a widget 1: (4) ting to vals uring atznton. (The way we have boon dig itso fa). 2 {u)Uang koe a sate vans 3.10) Uang ne wdpets ong mae. (U) Widget Attributes {U) Tere etn of pts fat canbe Sel ithe ae afew fe important nes Atibute Description Dacharomé / b= ‘The clr ofthe body oho widget (09. rei “bue’ “eree’ “black. foceqroimd / f¢ The color used fr text pase, pay “The amount of pacing to put around the widget horizontally and vericaly, Without hese te widget wil be jut big enough forts content bordersitn Creates a visible border around a widget hestht, width Specs the height an width ofthe widget. isableforegroond When a widgets disabled, this is the color of its text (usualy gray). Dofaults “rorest” but also can use “disabled” or “activeDoc ID: e005, (U) Example 4 Lats ty mong Example 2 ung he yale metodo maka oral and he contin) mato tmnt or btn, sport shiner a8 rors eer aed «te atest) Ilneiitet™}st aes Ine wage” using boys beter» te uttotrot) cen configuretert=r oo batten’) avrtg configure abe. ae) sa toot to 90 eton poe) tt baton to oe a (U) Geometry Managers (Now tat wei tow create widget wee on sap Wo arangig tem nour wrdowt (U) Mere are ai wo oes et geometry managers: (Tres aoe tr-Pace-bu maybe ats nt rey (U) Pack (cc nn tc ng ge comet une Gis ta ‘Atsbute Description “on Can be X.Y, 0° Both. x does the horizontal, ¥ does the vertical cxpind__Falee means the widgets never resized, “rue means tho widgets eszed when the container raze. side Wich side the widget wil be packed agains (ToP, SOTTO, AIG OF LEFT). (U) Example 5 ‘The packgoomey manager arange wig rv 6 windows you ar puting hen For example, you Slt Sse wl pack you weg agaist nee ideo he weetoc ID; 6589695 ccm using pach gentry monger ‘ron enter Import = rors m0. parent = Franarost) I pacing wages top-don bettongarent ext at TS ELA). pace(80229 Dettongorent, textANK TO ASS") pca(FI0) Dettoiparent, text CATOH RETF U CAN) pack (81-8) bucto(garent est LEF') pucks LE) Butto(gorent, ext CERTIR) pac stae LEFT) Dettonsorent, ext MIDM) pocket) arr ace | root minionp) (U) Example 6 Lceneray, he pa geomet manager intel fr snp g's bu one way a make more compat gs ung pce group widgets geter na Fran and en ads tte Fave to your non ‘sci uring pack genety monger ‘roe tinterSaport © root =O frame plication aarorrot) Sp manlonn)a st (U) Example 9.2 (0) ange heya mation made it aso check we wee oir he patos 1. Chace wget 2 Rate wrdowarange 43 hat neonany ‘Butts haat wre th odo using the ntze method. Even your code sit win nthe oder ct hese steps, hss lhe ode you want think about hem ‘tron tntersmpore as sophation frame) las Aptiction +: taste Taner eample et say penton) prime ello Pere") et create naget(se) SEAL here» Dltonsef, texts"bllo"fe"lue"comandslt so hello) Settee. pak(sie- Let") sf qUT = autton(sef teeter! ,comandeeF gut) SF QUT pees ett) et _snst_(saf, master © toe) Frame _snit_(oefrster) farm St erste taeets() at ming foot 0) Tio > Apllcatton(aster-rot) Spemsinlcon() a (U) Example 10Boe ID: 6680695 SiMe we nave covers te tree ses of Gs tlecang wget, ararging hem aur wndow, and ang nconaly, lets ya mere compte sameDo ID; 6589695 fron tlnter Lepore © ass sopntention trae) Joplcaton — the main app for he Fane... 18 all happens here. Fee's safine sone Case aerintes et _snse_10tfsaetortone) Frame _inte_(elfsnester) ait cont set gra sie() ‘= habe 0 checheroord wth Labels of efferent clors {i cheers inn) et _cont sopra. stntet Laying ovt the mster eri 1 ese nore 40 Lines encune that the arid tater up the entire Saf waster roconfiguet Sit rterccolomctignre = how creating gr on the ath window for in ong( set main rou): seit reconfigure’, seh) for in onge(self maton 01): Slt colucontigre(), oP) sf erlaaticy a1" A) wien) te)eo ID: 689695 vie “bow et chechars sain. wine): Crevtes checternar pattem rid ayes coors = se eotorgenc) for rm png sef an rns): for cin rnge( sa man, c8 ten 0), a) arma) abel (setter tr bg (olers)) 1.gria(raecolmme,stieysel® AL) et main roots) eee the 22 of our window root ponety( 0060") toot ithe heesone Gal = Tt 8 ny CL") t= sophiation(anrer-rost) sop etntont) Si (U) Exercise 10.1 (() Weave ove af wh atl. We dor want distur cura soles puta new frame ontop. Note how cram ines up en he i makes realy e257 sae how feting te ror and column igrment was aaa rnepr an clamsDo ID: 6589695 ‘ro tintertmort = ass ppttenion trae): opicaten — the main app for he Fane. 18 all happens here. at's sefine sone Case etribtes ein rows ‘Simin ates et _inse_(etfacterstoe) ane, _nit_ (senator) sea eont sel arid tee) = mh o checerdoard Oh Labels of ifferent colors fear checters meinen) feat ned tramea() et cont sentria stzecete): 1 Rayng out the mater grid 4 These nt 00 Lines ensure that the grid tones up the entire Seat master rewcontigure(®, wet Saf-nester columeontigure(oneetaht) for $n ange sat sain rent seit rmeentigrey were!) ‘or jin ooge(sait sims te): fe. cotomcontigure(}, ele) seat gria(etiny salt at) et colrgensei)oc ID: 6589698, ‘el re a “Soe et epecsars sain nse): restes 1 ehecaernard pattern eid nyect ors = sn. catorna() for 6 in range(sef-maoain rote): for cin rngesnif mata, 218: at = "Tew (@), GY formar) 1 Laer(etttse- top nee cates) 1. arian colnne,seeny te AC) at a8 fronton st tran tran selt gre") {F-fraehria(rw-® comm rouppan-9,columepn-sthacea AL) et maint root) oot pronetry( 0060") Foot tele hesome al — 1848 wy COO") t= sophiation(earter-ost) spp nino) $6 pone om “stn: ain) (U) Exercise 10.2 enanon pu ween re Nae aie crn ek made saan wan ah ners een oi wee ung i ower) Exes fo oder rome was bln a we ee a tet widget dest ok od arymere. Whethapeened!? Mw acaly ward area! wdbel wa shoul we do sero?ee ID: 6689685 fron tinterLmport = ass ppteation trae): opication — the main app for the Frame. Tall happens er 4 at's safine son Case etriuter et _tnse_s0tf astro) Frame, init_(set ester) at eon elt grid ste) 1 tone 0 ehecheroord wth Labels of oiferen coors fat checaars main) et cont st rissa rn tying ov the master eis Sa master owcotigure(, wet) eat sastercolumean igure) 1 new eeotng grit on te nen wna ‘for in rng se ata rvs) seit rowontigrey weg) ‘or Sin rong snfwatan-15) {ait catumcontigure(3, weigh) sat aratetncnyanlt at) et colorgensel):Colors = sel colorgen() for 3m anges matin rou: for in range( set natu.) fet ee (8), (Porat (re) 1 tapertoety text te open (0165)) {Lgria(nawcalamcesteyeelh AL) et at franc: 1 frame 3 lca say 80 show ha £0 mp ut 8 arid sta frame tram setstgra") Sef. front gria(rw, column, rwspasicolumspae, stcesa. ALL) SAL front teen orton fran!) [Sei fronet tent pac enpad Troe 106A) et min reek ‘ot prone Buca ot ttle eso Gl —- TE wy C00") ‘= Applicationnartr-root) s.msincon() a (U) Exercise 10.3 Las some anoter ame.oc ID: 6689695. ro tlt import © elas tpcatton rae): Aiicaion ~ the man app forthe Frame... TE ALL haps her. et _inte_(stesmstestone): Fron nde (oe mater) Seif eon st eid see() ‘= mabe o chachrtood wth Lobels of sfferet colors Sat. cnekers main win) Sef a trame() a a8 fran) et cont se grid saatsnty: Pn laying ost the master ri Sed waster owcotique(, wet) [Sc mater columeantgure(ovwesy) for 1 in eange(se maa rows): Sei romonfigwre(t wee) for jin ronge(set asians)! et columeoeigure(3, ele) ser gratsticayselt at)Pee: 682607 congas): Cenerator function that altrnrtes beaten re and Bue Creates» eneceernard pattern grid nyo ors = sa colorgen() for + in ranma ros) for in range( sa main, 2): fet tan (), GY format (re) 1 apr (set cont tetop stealer) 2 artarowe cole, stehyslF AL) et ats franz eee are to put att tn seat tranetron( se bere") Sa frame. rion coh rowspane8 compan, stikosekAL) Sat fre. font soto (set frame). Sif frame. est pcklenund rue, 01-4904) ef ats franentent) Pereen frome! Sarat Fran(sefbearen’) Satan arierow conn ronan, column they: ALL) et mn rote) et rene s0n60") ot Sele Muesome us = TE any COL") ep depiicattonraster-root) sp-maintoon) vain "hatte wanted eu GUI to do someting when we cick on ur gen tame? Frane doesn law uso Set comand Bu wal Hopes ots We can in ou rae fo an ve.Doe 1D: 99555, Binding to an event (U) Net wtp have a om optn, bith des man youcam ist wt am Aso, ee may be instances when ou want response fom fhe user han 2 muse {Sek (whch swat he bey comand cin spends} hese tance, ou ware Bnd your gto fe appotae ever omat edie (optona = event ye - dtl eotona) Ferexample: bston media's one event pes Auten ond dlls oe. Thea he ever he etme ban was ced Forth ig mouse btn, you would we Coneon event pes: Sion, sutontleie, Kyselease, Keypress FocsIn, FaCwOu, Lane (hen he mate leaves he wig) an Yosebee! Conmon medi: A, Any (Used Re cin keres>} Contr, Doble (Used Bem eset) ‘Conmon datas: These wil vary wiley Base on the evertype. Most convert you wl sac th key or Keres 0k cteyres-F2> Exercise 10.4 Letina geen tam oe left mouse ion ck nd pinta corn lhe ck. Since pining oor olabook oe command ne ol ry unk for GUT es ‘he apy We coosnats ero okDoc ID: 689698 fro tinter import = ass popeation( trae) Jeplicaion — the main s0p forthe Frame... KC AIL happens har. ets define sone Cass attriates ‘main eals| xt _inat_(seif astro) Frane_snte_(eltsnaater) teat cont seit aria size) |= hake 0 chechertoord with Lebels of efferent colors feat checters main in) sent ad_franen() feat ade fren) nt cont seit grt staat rn toying ov te master grid if mater rowenta, weet) Sat ster. columeenfgure( yeah) {or in rng( sa stan rw): Seif romentigire(y weg) for jin oongetsaif maa ol): Se colummcotigure(S, wee) snr gratetchy ant at)Dow SF catergen sel: enerator function that alternates betes rad an bie ‘etd “re ald “the et coctors main wins) renes »enecerboard pattern ar leyout colors» se. colorgen() {for An range sef ain ro) for in ange( sei satan, 8: tet Ten (0), GY formar) 1 net ett texto neeeales)) AL grla(roversclamse steepest ALL) et att frames) ‘ad frame wt 9 ent area to pu st An set traet rans ge) at franet rid cso rowpan a0 trae ent wter (set fre) Sa frame tent ok oxpand- Troe 10-84) clumspan-0,etekst AL) et aos fromn(set) reer rane! Sat fraed = Frame(seFbe~aren’) St -franes rdw cal rowrpnscolumsgess tekst ALL) ‘t fraeh nd auton "ee fra ander) et fran pnde(se event): sigs rane 2 ehaed at ()()-fOret(evnt.xovent.y) eines) dete) ‘sere (0, 8) at sain: foots) oot gronetry( sn") oot tle hmesone Gl — TE 8 way C08") ‘= Aoptiationeaster-root) sop-naintoo)oe ID: 6589695 Reso eras (U) Example 10.5 Creat Lets ad some butens now Eventhough btn have 2 comand strut, canbe cy fo pat information about a itl baing cick using We wo ave owe 8 ‘Sarl ncion leach uted, tus ay ig eo we an acoge he buen sr Hom ha event Note when we arange he buttons we have align hom by Incimers bee sce hy span fee clues.oe ID: 659695 fre tint daprt * clas sppitation frame) opication — the main app for he Front. TE Alt hapent bere ets define sone Cass ttrinter et _intt_(senasteratone) Frame _inke_(e6 stor) Seif eof st ait sse() f-- mone 0 ehchartoans wih Labels of sifferet colors Seif. chechers ain win0) sa 38d froel() Se 08d frome) Se net atone) et cont set gr szatsty 1 laying oot the master er Se mater rowcotigue(, wip) Seif mster columeontigare(ovme) for in rnge ant sain rove) Seif rowonfigre(y weg) for 3 in range(set sainain eas): eit columeonigure(s, ele) sexraratstiaysel? At)oc ID: 6680655 et colorgensai) ela ed? ae “ue” et enecears. sain wins reaes @ cnehersoues patter er Layee colors» self coloreend) {for An emg(sef main rows): for in ange(sei satin): tee een 0}, a) formate) 1 Uapertoet texte open (ot) 1Lgrid(noercalumeestiey-sel AL) et na fomen(sa saffron rane") Sat frome. rian conn ronan colimspan-,sthkoseh AL) Bat frame entered frame Sat frame. tor pokenand rae FLL AOTY) et obs traverse) Sat franed = Frae(sefb'ereen") SA. fronez gria(rwcolume,rwspass,columspans,stchy-sel AL) Sat framed bind ebatan- sl frome handler) et frame. pater(selt, event) "ands events fron fran!" segs Frame 2 elihed ot 0) {)"formt(oret xen) princes) hie troes.tent melee) (Saf front textn Snsert(00, m5) fet 204 buttons sear baton st) ieton labels = [ee Blue Green, aa’, yellow)oc ID; 6689695 for c,0t in enumarate(utton Jets PS naton(sety tee 80) toteipes of thee ertd(rwcotamec),colamspan,sthcysel6 ALL) bind ctuton-1 sf -buttens ander) sche ution hie aero) teton licked - evant. atdget{‘tent*} prtnibaten lice) se frames text dete 0, 80) 0 frame ext Inert (OD, btton pis tile('sele") lest) The matplotlib object-oriented API ‘Tha man ida wih stot ore prograrmings to have obs whch re can apy eons and acral oo! of rogma saes shou be bal (uch a the ATLABse At) Thee! advantage is aproach becomes appre when more Manone ores resid er nen are crane more Manone BL Tosa be ctpcnenig AP. we slat vary mach keine revs xa, instead of crating anew bal re starc, wera refrece the ety cented are Indarce nthe ig varabe snd om we senleanew sarc sser ung he nodes methane Figre Gans nence Pe ig ot. tipre() ven « Fa.a taro, 9, 1, 0.31) # Left, botton, wet, eta (range @ t0 2) sven plete. 9s“ sop set abel") boon set gle Y > oon settee ee) ough abt mor odes vated, He advantages tht we now have fl contol of where the ple aks replaced and we can easy add mare an one ax the ge:oc ID: 6589695. Fig» mt. Fiera) ate figure eran pate 9 rom se abel") rot sect y") from set tele Tele) ero plot ee “8D roma set ete 9") ram set abel) froma set ttle inset Tite’): Tose gue oa ie, we cn ue te saat metad nthe gue ae fg. seep Filenne g") seaborn — statistical data visualization ‘Sestomis a Pion viustaton trary bated on matplotib. prow highevlinetce or raving tractive sal gris, Homepoge for he Sobor project io san al -myasion/sauamisstany bokeh — web-based interactive visualization ‘Baten a Python rtoacev Yeuakzaton ort gots modem We browsos fr resort. sg! so pro logan concise corstucon of novel gaps iho sh of sts mt also aver hs copay wi gh prtarmance nary ovr very age or rearing datasets, Soke an ap anyone nha woul eo Gch 2 easly este Inuractre pts. dastonass and dla apteabons. Homepage for Boke inokeh tact NumPy Arrays NanPy ays (uy. narry rte fundamen data ype in Num. They hae: + shape + sr dlomont pe cao type Fo ena: tes wo zeros(shape (MA), type fleet)Doc ID; 6589695 areeros = m.2erosoo) Print Corrzers.dtype, areeres she) arverst = 9p 2608((8,20) (runorrerosz. type, atrzers2 shape) ‘sine NanPy ary coreszoning he wo rerio! mat setae NumnPy hes bot a genre N-dnansonal ry and spc 2 nano mati ata ype, NumPy rays may have an biary mumbo mens. unPyarays aver vectra matemstal pean, ar = to. arange8) reshape) eine rigina”) pranior)| rin) print “eenenaise comptes") printrss) 28) rina ctype) arr eanapetasz)89°2) 99 are reshapts 294972) £38 see-resmapetsy 7) Array Shape Mary say craton facts lake shape parma: For 1D aay, he sage canbe an lager ‘rin. eros stapes, aype-np s8i6)) Fern ays, te shape needs be ven as ape print(co. sero stapes (42). typeset), ents) ‘print rnrnp.eros(shape (2) eypetiest))) prin. sero stapes (2.2.22) Array Types -Aaraye have a space pe orl aesocatad ements. Every elm n Be aay shares tha ype The NunPy amndiogy fis ype sap. The bas pes ‘aroun oa. and complex. The ypes maybe made by a mune indeaing hor sze nbs. Py ulin ypes canbe used as a caesponding dye, Noe, th {ovorc NPy types ond wih an udrscor "1 terete the nan Fone Pt banNumPy ctype boot boot. int ont float npsoat_ complex p.complex Hor isone example o specthing aay: te =o 2ees(shpe(5), Aeypenp flat.) # Nity default steed Fo rincare, "=, ar. te) Wat au, not array = op-aran s9200e7090097122, 3, pene e836) re flestae(an") Conckine deta ype rane etypenin ctype Now we tho juste moment define ne quick per ncono show us hese des in a prety format et au arry(er) rine array of X83" = (arr shape, arr. type) brintor)| Array Creation Num rosa number of ways coat an aay. np.zeros and np.ones = wp.2ees(shape29) imp erga) rit. onesshapes(2.5)) ove. arr = npones(shape = (22), dtype-int) Sip eraytene set) np.emptyDoc ID; 6589695 ESSE reingauck because simpy requests some amount of memory am he cpeaing estan and then does ating with Thus thay rene ‘by seeetyIsunnaaed Conse yours wamed.m.eerty Vet uel you ow Youre gg lly al be (use) elaen our ayaa. ¢ fogs tte eticand pow vit iy tthe one dffrene wate) tres rocenty(ohpe (2°). ype int) ‘inp araycer) np.arange sv.arage ganeries secuencs of numbers tke Python's rage Buin. Norio top ates may lao unexpected resus othe cases, you may preter. Lieace and stobelon. Fora quck and mest practea— cussion ofthe pons of fotng pot approxtatns 868 issn.) + asje valve stopping pont + to vans ae staring pot anda stoping pont * toe values ara at sop, anda sp ze ‘swith sane ening pont notin rina ar $e" np. aroge(e, andveep) of ronge(ston) yf envsap) © cf rant ston) 72,3), envy) endpoint exuded Print neg. reps As" © psarange(i, 8) 1-9), np. linspace snot, 8, ure) grr nce NUMPT nae fete sent on [BEGIN, END] FBEGILEND Une ors range an vse 8 ‘Scns wae BEGM an ERD econ ee). rit ttins are neni", ea) i ean) 3,3, sap Fant simpeents, 3, 2), Seep telnet, st ne) Diagonal arrays: np.eye and np.diag reser pads an aay wh shape LN and ones on dan an NaN ety mat) Print op-ee()) Arrays from Random Distributions. this conmen orate arays whose laments ar sale em aanéom distiulon For he many epons 08 + tlpnpandom)Dow 10,6009 hiform on [0,1) prin ‘Untfore on (0,2):") ‘in. arayvor.ranen( (399) ‘Standard Normal ‘cecando has some redndancy. also has Some variation calng conventions. ‘+ stuntordsorma takes or ue argent rai (whch ser carmen ose mca) akan arguments where nis he number of dmenions in ores print. normal 0.20") ‘inp aresy vor standard nore (2,5)) rien) ‘inp, aren nor randn(25))# ne ple poroneter Arrays From a Python List...and a warning! ‘us ao gone orate NunPy ays rom Pythons an tuples. le ths = @ nee cana, rember hal stantatng 2 Python a a ake lay ong cme to ‘recy ing NunPy bug Bock. Oe caiarrs and erbles wll goreray ve uel sus. ‘imp. ereyo.array( (is 2 3D) rim) inp 2raye.arrayiL.8, 28.9, 5D) ‘imensonaltys marta wit nested: mp. arayo.array( (Cs 2 Ie tes tly rim) in erase. arry( {C05 21, Bea eli» Accessing Array Items Indexing tas uy says may be accessed using # sng index composed of mute vakes SiooutDoc ID; 6589695 tr « mvaragn()-rehape yt) rendn.rondin(3, sizes, 6) prart-ebe array") rineare,enesen) ranean 3.2) 2°, are(s.2}, enéveen) prume-ine [3] 2°) erie enbsey) 1 nan-atontsc, crests © view of ar(3) then ineues ‘ate hot copy printing GB], aPeDIED ‘Comparision nto need Python et ist [Lse(eoH for row in aerd primes) princaustto302D pene (osst(21) Slicing Wo can 400 uo song sect entre ow and cola at nes {état sumo. sce 0-nogha.pg) Ketdtiet_nampy._sce,02-nospha.pa) Important Differences Between Python Slicing and NumPy Slicing + yan song retuneeapy oft orginal ta ‘Changing the sce wont change te og + NmPy song reuns a ew fhe orignal data hanging the sic wil change he ein ata “ho Nunes Pape haba tor formation.‘Boe ID: 6589698 prareaeay) printer) etn \aceassing 9 roe") sme sray are? 51) rant \accessing column: (ep erayret I) the cae, atthe sepa, ol Prine Cs", rl s2), "as Shape ana) teeta} shoe) Boer n ing tat mumercal indexing wl duce the dmansonalty of the ary Sng fom snsex 10 Index an be Seda Keep thal dimension fou aod etn "se dinensn:”, nde” *) me svayare2n 3D) rintrhept dimensions", ene” *) See svayare2y 1) Region Selection and Assignment ‘ate shes 25 prt an index. can Sle region outa an aay rina") primer) print subarry:") hep sray (ney 222) Sos are ahays vows of he undring aay. Thus. mthing tam mdf te unering aray te =o aranges) reshapes) rinc‘oven laments (oto indies) of first roe") Primero, 21) # Select every wther element fro fst roe srr{0yi2] = 1 updte ts done place, m0 ony rinc"\afteraesinging to tate") printers) Working with Arrays Maths qt sipto—ae Piss part f he reason hat using NunPy aay cn scan sry numerical code. The gna pattem ary OP scalar oscar OP aay) apps (OP (wth ne clr vai) arose elorent of aay.oc ID: 6589695 Pee 1 aecrse ops (ef. mptegteoL ont, et.) rine? are |) ae) prime” ee BS, a 6) ore: arr te 1, ete. for i-plce ert parang) primers, primer, Elementwise vs. matrix multiplications Nun aay and matrices ate rela, it ight een yes. orange). reshne (28) mp. orage0. reshape?) tev) Print "Ceneneatse muleotication: 8 * 8) Prine *b, eneeeo) rin "or procs moot(7, Primi. sot(a ts), even) Prime prouct ay an arry atta: 2.7.10)" prima. cot), d-vsen *) rt, nat = mp ateins), ro.natese(o) rina, ast» mosnr( a), naseteia(h)) prin oat") primo) rit st") prin(oot, enoven) Printer prouct F matrices: amar * tat“) Prin{ont. 1 ort, enswse) frin"ot practi Pyar 3.54: rina. pot)oc ID: 699695 Some Additional NumPy Subpackages + tft — Feet Four tanto + ro.peimniat — Orthogonal plans, spline fing + pp.inalg —Lneor algebra + atety, ete, faves, tvs Isteg, norm, ar nd mum — Cetandadrry mat neon + pcrandon — Random number generation 1 ea, gama, ont, hpermoetri, Legperma, tors, gessen, wfor, wtb + many oer, you need NunPy robb has it FFT sap inepae(e, 120, 4090) sign <2 mpain@ S29re) #3 He 6S rpsint = 28r8 5 + merdentt)) Fete Fer aby. tf. 4g) fru fe. rte ee} ‘Sheed; It ie(, Hh ple sow) Pe plot ereess Ps For one-stensna reat puts oe con dscard the negative frequencies fer ae eet) Hefetregaignl size, ef] - ef) ple plet(fres, FF}: alt eli(-8.2, 1); “Tenting speedup of dlecarting negative frequencies Fer abeop. tf. ¢4estent)) frogs no Fe eftregsenal size, *(2] - (0)) Fer absnp. fF. rffestent)) frogs = to FR retre(siznl size, eC) — 01) SciPy - Library of scientific algorithms for Python Te SP ammo 6 ofa wey Fano mIBSmAnsn rs roe Be manatee pte Sona ae oes rat SaPy coverCe ~ owt ea : accra : antenna ry 1 Sone em ‘cena on : Staten ST pee ttn rr teh cemeaea tnt tea ms atom nnd a ns trap To access he SiPy package na Ptten program. we star by imparting everthing rom fe spy moder ely impart the subpackages we nee. Fourier transform Fourarvanstoms ate one fhe wives eli computational pryics ey appear ovr a over again n ferent conto. Sy proves intone accessing Stes [= [281 try fem Nt an elt an wel ested FT Harry tien FORTRAN. The SeiPy API has fw adstonal onvennce foc, bt oer te APTS oy led the cra! FORTRAN ery. Touseine ¢rsack modieinapyhon progam, nde tung: port sein. fftpck a8 opt Fev absent estes) frogs «gfe ffteea(stpali36, £13] -*(2)) pieplo(e, signa); ale alin, 3 pl. show) ie ploetsteasy Fr NumPy FFT vs. SciPy FFT vs. FFTW vs. MKLFFT Which FF ay sos you use? you wentoure he MAL. you must use NunPy. ‘The det stato of NunPy and Sey use FFTPACK FTW face an FFTPACK, and oe ator tren MK. Installing PyFFTW 1 tna FTW + sptgptintt thn et 1 yom ta ve 4. ota ytDoc 6689605 Interpolation Intarglaton simple and convnintn ScPy: The Inte function, when gon araysdescrbngX and dat, uns an jc that behaves ke 2 furcbon at canbe ead for ator vai (nthe ange covered by) rears he coresponing pod value ‘mort scipysnterpolate a spinter at 00) tien rosin) = mesarangetoy 1) ro me dinepee(o, 3, 19) sas = (0) + 021° ap. rando.ranén(a(a)) # simulate meosuronert with noise sored = 0) ‘imaesntrpalation = epinter interes, y.mea2) arg” nea interpation(2) ‘ibe imterplation» spnterSntarpta, y-nas,wing-‘ue") seiner” eabie interpolations) it, ox» pit. subolersFestze (38,9) Zetec, yeas, ‘bey label. "nelay 48a") Splat, yretly "t,o, labeler fnetion") Sepia, beeline iter) Se plete yinterp, “f, Lbeleabie ote’) Se epee (180oe ID; 6589695 COMP3321 (U) Python Visualization pecs (U) This notebook gives an overview of three visualization methods within Python, matplotib, seabom, and bokeh, Python Visualization “Tis notebook wil ge abasic overow of some oto vale data. Much of his male's developed by Continuum Analy, based ona ‘uo eeatd by Wesley Emenoker. First, install necessary packages: # fun this fon LABBENCH import ipydeps packages = [‘natplotlib', "seaborn’, ‘bokeh* , ‘pandas’, ‘numpy', ‘scipy', roloviews' } Spyseps. pip(packages) import nunpy 28 np import pandas as pa Visualization Choices “There are many different visualization chioces within Python. In this notebook we wil lok at three main options: + MatpltitDoe 89608 on + Bokeh Each ofthese options are built with sigh diferent purposes in mind, so choose the option which best suits your needs! Matplotlib “Matplotib is an excelent 2D and 30 graphics library for generating scientific gures. Some of the many advantages of this library incude: + Easy to got started + Support for LaTex formatted labels and texts + Great control of every element in a figure, including figure size and DPI + High-quality output in many formats, including PNG, POF, SVG, EPS, and PGF. + GU! for interactively exploring Agures and support for headless generation of figure fies (useful for batch jobs). Matplotib is wel suited for generating figures for scientific publications because all aspects ofthe figure can be controlled programmatically. This is important forreproducibity, and convenient when one needs to regenerate the figure with updated data or change its appearance. More information a the Mania web nage. To get started using Matplotib in a Python program, import the matploibpyplot module under the name pl: Anport matplotiib.pyplot as pit 4 This Line configures motplotltb to show Figures enbecied 1 in the notebook, instead of opering @ new window for each 4 Figure. Nore about that Later. If you are using an old 4 version of Iython, try using “Spylab inline’ instead. Dmatplotlib inline The matplotlib MATLAB-like API {A great way to get started with plating using matploti is to use the MATLAB-tke API provided by matpotib tis designed to be compatible with [MATLAB's plotting functions, soi you are familiar with MATLAB, start her. space(®, 5, 100) print(y(e:10])oe ID; 6589695 ple-figure() plt.plot(x, ys "#") ple-xlabel("") plt.ylabel("y") plt.title('title') plt.show() plt.subplot(1,241) plt.plot(x, y, ‘ro-") plt-subplot(1,2,2) plt.plot(y, x, 'e* The matplotlib object-oriented API ‘The main idea with object-oriented programming Is to have objects to which one can apply functions and actions, and no object or program states ‘should be global (such as the MATLAB. Ske API). The real advantage ofthis approach becomes apparent when more than one igure is created, or ‘when a figure contains more than one subplot. ‘To use the object-oriented API, we start out very much lke in the previous example, but instead of creating a new global gure instance, we store a reference to the newly created figure instance in the ig variable, and from it we create a new axis instance axes using the add_axes method in the Figure classiinstance #ig fag ~ pit Figure() graph = fig.add_axes([0, ©, 1, 0.3]) # Left, botton, width, hetght (range 6 to 1) raph. plot, ys ‘r") graph. set_xdabel (°x") raph. set_ylabel('y") graph. set title("title’); ‘Although a bit more code is involved, the advantage is that we now have fll contol of where the plat axes are placed, and we can easly add moreee ID: 689695 Fig = plt-Figure() graght = fig-add_axes({9.2, 0.1, 0.8, 0.8) # noin oxes gragh2 ~ fig.add_axes({0.2, 0.5, 0.8, 0.3]) # inset oxes # mein figure raph plots ys 'e") raph. set_xlabel(‘x') ‘grapht.set_ylabel(‘y') raph. set_title("Title\n") eraph2.plot(y, x, ‘e! ‘eraph2.set_xlabel("y') ‘graph2.set_ylabel(x') agraph2.set_title( Inset Title"); Plotting categorical data Note: this works in matplotib 2.0.0; in version 2.1, you can enter the categorical data directly on many ofthe matplotib plating methods.ac ID: 6589695 # (attiaLtze our data here, turning this into 9 List of names and a List of counts aca 420, ‘oranges’: 15, "Lenons": 5, "Lines": 20} 4 First have to create numeric volues to cover the axts with the categorical dota N » Len(names) 4nd = np. arange(N) vdth = 0.35 4 this will make three separate plots to denonstrate fig, axs = plt.subplots(2, 3, Figsize-(15, 3), share) axs(o].bar(ind + wath, values) axs[1].scatter(ind + width, values) axs(2].plot(ind + width, values) 4 here we'Ll space out the tick morks appropriately and replace the numbers with the names # for the Labels for ax in axs: ax set_xtlcks (ind + wlth) |x. set_xticklabels (names) ag. suptitle( “Categorical Plotting") pit show Fie) In Matplotlib 2.1+, we can do this directlyDo ID; 6559695 ‘ron bokeh.sampledata.autonpg import autompg 25 df Fig = plt-Figure() graph = fig.add_aves((0.1, 0.1, 2.8, 6.8) rumvehicles = © raph. bart Gf[ ‘nane"].value_counts()-index{:nun_vehicles], 4 ‘nane"].value_counts().values[:nun vehicles] > graph. set_title( Number of vehicles’) sgraph.set_xdabel (‘Vehicle nave") raph. set_ylabel ("Count") pit. show) To save a igure toa fle, we can use the savefg method in the Figure class: Fig. savefig( “Filename png") ‘The real power of Matplotib comes with plating of numerical data, and so we will wat to delve into Matplotib further until we talk more about ‘mathematics in Python, seaborn — Statistical data visualization ‘Seabom is a Python visualization library based on matplotb It provides a hightevel interface for drawing attractive statistical graphics. ‘The homepage forthe Seabom project onthe intemet is here Inport seaborn as srs {nport pandas as pd sns.s0¢()oc ID: 6689695 ig = plt-figure() ‘graph ‘graph Hes({9.1, 0.1, 0.8, 0.8)) # main oxes wes([0.2, 0.5, 0.4, 0.3]) # inset axes # noin figure grapht.plot(x, ys 'e") graph. set_xlabel("") ‘graph. set_ylabel('y") ‘rapha.set_title('Title\n") araph2.plot(y, x, fraph2.set_xlabel("y") ‘graph2.set_ylabel(x') graph2.set_title( Inset Title’); ‘More examples! import randon GF = pd.bataFrane() F(x") = randon.sanple(range(,100),25) Fy") = randon.sanple(range(s,102),25) F head() Scatterplot sns.mplot('x'"y' ,data-dF, #it_reg-False); Density Plot ns. kdeplot (Fy); Contour plots sms. kdeplot(dF.y, €F.)5Doc ID: 6669695 stribution plots sns.distplot(4F.2); Histogram plt nist (ax alpha= ns. cugplot(af-x); Heatmaps sns.heatmap((dF.y, dF.x], annot-True, fat="d"); Bokeh ‘lah ie @ Python interactive visualization brary whose goal to provide elegant graphics inthe siye of D3js while maintaining high-performance interactivity over large or streaming datasets, Bokeh is designed to generate web-based interactive plots, and as such, it may not be able to provide 18 fine @ resolution as Matplti. The homepage forthe Bokeh project onthe itemat is he. “There are multiple options for displaying Bokeh graphics. The two most common methods are output_file() and output notebook() ‘+ The output.notebook() method works with show) to display the plot within a Jupyter notebook + The output ¢ie() method works with save() to generate a static HTML file, The data is saved withthe plot to the HTML fie, In this notebook, we will focus on output notebook() ron bokeh.plotting import figure, output notebook, show From bokeh.resources Amport INLINE ‘output notebook(resources-INLINE) import holoviews as hy hv.extension( bokeh") Fest, we will make input. Bokeh plotofa Line. The Line function takes a Ist of x and y coordinatesDoc ID: 68969 4 set up sone dota import nunpy as np x = np-Linspace(®, 4mp-pi, 100) np.sin(x) splot 0 Line plot = Figure() plot-tine(x, ¥) shox(plot) Styling and Appearance ‘The ‘line’ above is an example of an object called ‘Glyph’. Glyphs are made of ‘ines’ and ‘filled areas’. Style arguments can be passed to any glyph as keywords. Some properties incude: + Line properties: Line color , line alpha, Line width, and Line dash + Fl properties: #311 color and #311 alpha Bokeh uses CSS Color Names. Here is another example showing styling options:ec ID: 6559695 X= mp.Linspace(®, 4°mp-pi, 100) y= mp. sin(x) plot = Figure(titie="Sine Function") plot.xaxis.axis_label='x' plot. yaxis. axis label- "amplitude" plot. Line(x, ys Line color="blue", Line wiath-2, legend 'sin(%)") plot-cirele(x, 2*y, egend>"2sin(x)") ALine_dash 1s an oribrary Length List of Lengths falternating tn [color, blank, color, .-.] plot-Line(x, np.sin(2"%), Line color="green, Line_dash-{10,5,2.5]4 ine _width=2, egend="sin(2x)") shon(plot) Charts Bar charts ‘The Bar high-level chart can produce bar chats in various styles. Bar charts are configured with a DataFrame data objec, anda colurmn to group. This column wil label the x-axs range. Each group is aggregated over the values column and bars are show forthe toa:ic ID: 6689695 ‘from bokeh. sampledata, autorpg-head() stomps {aport autonpg as autompg hp_by_cyl = autonpg. groupby("cyl", as_indexFalse).agg( "hp": np-mean}) p = Figure(title-"Average HP by CYL", plot_width-60e, plot_height-400) P.vbar(x-"eyl", top="hp', widthe0.5, sourceshp_by_eyl) shone) Categorical Bar Chart Fora categorical bar chart, we sil use p.vbar as above, but the top value wil be the counts forthe items in x . For the below example, we used the Colunoatasource class from tokeh.rodels to actualy tore the data, which we can then pass in p.vear under the source keyword, We ‘also imported a color palette from bokeh.palettes to use as the color palette, passed in with the color Keyword in our Columoatasource. (Note: ‘when using color palates, you need to make sure there are enough colors in the palette o cover all he values in your data.) ‘from bokeh.nodels import Columbatasource ‘fron bokeh.palettes import Spectralé fruits = ["Apples', "Pears' ,"Nectarines', "Plums", "Grapes", ‘Strawberries"] counts = [5,344,244,6] source = Coluanbatasource(data-dict (fruits=frults, counts=counts, «color-spectrai6)) p = Flgure(xrange-frutts, y_range- title-"Frutt Counts", toolbar_location-None, tools: 0,nax(counts)+3),plot_helght=250, z) p.vbar(x-'fruits', top-"counts', width-0.9, color-color", legends"fruits', source-source) P.xgrid.grid_Line color = None p-legend.ontentation = *horizental’ legend. location = “top center” show(p) HistogramsDow soos ‘Simple histogram using Holoviews Using holoviews, we can easly create a histogram on tp of Bokeh with the hy.tistogran function. output size-150 Stopts Histogram (Fi11_color='#CDSCSC", Line_color="black’) w-Mistogran(np.histogran(autompg[ ‘npg’, bins=20), ka 7, 8, 48, 45)) hist More complicated histogram using native Bokeh syntax We can also use the qué method. In this example below, we're actualy using the np.histogran function to create our histogram values, which fare then passed into the quad method to create the histogram. Anport scipy.spectal Ainport umpy 2s np ‘from bokeh. layouts import gridplot P = Figure(titie-"Normal Distribution (muse, signa0.5)", tools-"sa background_f111_color="#E800CB) su, signa = 0, 0.5 ‘neasured ~ np.random.normal(ms, signa, 1900) hist, edges = np-histogram(neasured, density=True, bins-50) X= mp.Linspace(-2, 2, 1280) P.quad(top-hist, botton2, left-edges[:-i], rightvedges[:], Fill colors"#036568", Line_color-"#933649") p.xaxis.axis_label='x puyatts. axis label-'Pr(x)" show(p) Scatter plotsDoc ID: 6589695 ‘Soutput sizenise scatter = hu.seatter(autompg.loc[:, ['sps", ‘hp'1) Curves ‘output stze-150 accel. by.hp = autonpetrounyho", ax index-Fase). ag aces": mp-nean) opts Curve [height=200, with=s00, tools ‘opts Curve (coler="red", Line width=1.5) rover" 31 curve = hv.curve(accel_by_hp) Spikes outpue stze-150 spikes = hv-Spikes(accel_by hp) spikes Using layouts to combine plots [As simple as using + to add pots togetheres as | ‘Soutput size-120 ‘opts Curve {height:269, width-00, xaxis- ‘Sopts Curve (color="red, Line widthei.5) bottom] ‘Slopts Spikes [height-200, width=i09, yaxts="left"] ‘Siopts Spikes (color-"black", Line width-0.8) Aayour = curve + spikes ayout A taste of advanced Bokeh features Bokeh is loaded with wonderful features. Here are two final examples with no explanation, See tnesenolebooks for addtional Bokeh information. ‘fron bokeh.sampledata.irts Import Flowers ‘lowers. head) ‘fron bokeh.nodels import 8oxZoonTcol,ResetToo! ,HoverTool 188 Add a-new Sertes mopping the spectes to a color colormap = {'setosa': ‘red’, ‘versicolor’: ‘green", ‘virginica’ ‘lowers{ ‘color'} » flowers{ species” }.map(Iambéa x: colorwap[x]}) blue") ‘tools = (Boxzo0RTo01(),ResetTool (),HoverTo01()}] plot = figure(titie = “Iris Morphology”, tools-tools) plot.xaxis.axis_label = "Petal Length* plot.yaxis.axis label ~ "Petal Width’ plot. circle flowers{"petal_length"], ‘flowers{"petal_wiath"], colerflovers{color"], #assign the color to each etrete #411 alpna-0.2, s820-20 ) show(plot)Bo ID: 6559608 x= mp.Linspace(e, 4°mp.p, 100) ¥ = mp. sin(x) plot = Figune(tools="reset,box_select,lasso_select,help') plot.cirele(x, y, color="blue") show(plot)Doc ID: 6589695 mel Module: Pandas ‘psec gr ear ape by Tn ates Sas (U) This modules covers the Pandas package in Python, for working with dataframes, Pandas Resource & Examples . (Wot: this was modified trom the Pancismenism natatock by[—Jon nbGalery,) This resource should help people who are new to Pandas and need to explore capabilties or lear the syntax. 'm going to provide a few examples for each command | introduce. Its important to mention that these are not all he commands availabe! ‘Aso note that Pandas documentions is available in DovDocs. Fist wellimport and install all necessary modules. Inport ipydeps rodules = ['pandas*, "xlrd’ "bokeh sraquests", “requests_pki", “openpys") pyseps.pip(aedules) “pcs the standard abbreviation for pandas, and "np" for numpyoe ID; 6689695 Amport math import pandas as pa import nunpy 25 mp stnis 4s only included to give us a sample dotafrane to work with ‘from bokeh. sampledata.autonpe mport autompg as df Creating a DataFrame ‘The vary basics of creating your own DataFrame. | dont find myself creating them from scratch often but | do create emply DataFrames lke seen @ few times further down in the guide, create Empty OatoFrane Object 41 = pd.vatarane() ‘this is the very baste method, create enpty DotaFrone but specify 4 colunns and their names sou can also specify datatypes, ‘index, and many other advanced things here 41 pd.bataFrane(colums-("Columt', ‘Colum2", 'Colum3' ,"Coluand')) create testing DataFranes (0, b, ¢), alaoys useful for evaluating nerge/Jotn/concat/append operations 2 = pé.DataFrane({{1,2,3],[32405]]» columns=List("ABC')) b = pd.DataFrane({[5,2,3],[7s4,5]] coluans-1ist(’B0E')) € = pé.ataFrane({[11,12,13],[27,14,25]], colums=Lise("xy2")) Reading from and Writing To Files Super easy in Pandas csv Lets write our autompg dataframe out o csv fist so we have one to work this. Note: f you leave the index parameter set to True, yout get an extra column called "Unnamed: 0° in your CSV.ec ID: 6589695 F to_esv(“autonpg.csv", index-False) Now reading itn is super easy. F1 = pd.read_csv("autonpg.csv") = 16]-reset_Andex drop: True) Concatenate DataFrames ‘Simlarto append, but handle large ists of dataframes well abe = pd-concat({2,bse]) abe Join DataFrames ‘SQL sh oin operations (Inner/Outer etc), can specty jin on index, similar columns may require specication ! Joined of = a.join(b,hows" Left AsuFFD joined_af Merge DataFrames Merges 2+ DataFrames with overlapping columns, Very similar to jonoe ID: 6589695 snerged dF = a.merge(b, IeFt_on="8', right_on-"0") serged_ at Iterate DataFrame erating s only ood for small dataframes, larger datatrames generally require apply/map and functions for efficiency ‘You wil inevitably use these methods at one point or another. Iter Rows ‘Access to values is done by index. rowsj0] = Index rows{t] = valves as pandas series (similar toa dct) ‘ows 0} = First column valve of row, can spec column rows{ {Colum} for row in of Sterrows(): Af counter > 15: real print (rows ]-keys0(01) nt(row{:]{'nane"}) Print(row(0], row(2]T0D) IterTuples Faster and more effecient, access to values is slighty different from iterrows (Index isnot nested). rowtuples(0] = Index rowtupls{t] = First column value rowtuples(2] = Second column valueBoe ID: 6559695 for rowtuples in df.itertuples(): Af counter > 15: ‘break print rowtuples[1},rowtuples{2},rowtuples{3]) Pivoting on DataFrame (Croate Excel style pivot tables based on specified crteria maste Pivot 4f.pivot_table(ndex-[ mp ‘name"]) head) sspecify for a more complex pivot table 4. pivot_table(values-["weight'], index-('cyl" ane}, aggfunc=np.mean) head() Boolean Indexing Fite DataFrame on Multiple Columns and Values using Boolean index "Note: The 8 inthis example represents ‘anc which might cause confusion. The explanation for his can also be abit confusing, atleast it caught ‘me off guard the frst few times. The 8 wll reate a boolean aay (of True/False which is used by the fering operation to construct the output When all 3 statements below retum true fora row, pandas knows that we want that row in our output. Te ‘and’ comparator functions diflerenty than "8! and wil hrow a he tru value forthe array is ambiguous’ exception. AF Loel(AF{"ey1"] <6) & (afl mpa"] > 359) head # the sane thing can be done with .query, for a more SQL-esque way to do it 1 just beware thet you can run into issues with string formatting Hhen using this method 4 query(“eyl <6 & mpg > 35°)-head() Crosstab Viewing Contingency table (also known as across tabulation or crosstab) isa type of table in a matrix format that isplays the (multivariate) frequency dlstribuion ofthe variablesDoc 69485 pe. crosstab(af{ cyl" ],dfl "yr" ]margins=True) Example using multiple options "Note: This fs an example using a combination of techniques seen above. ve also introduced a new method largest ‘0p Number of Column! Unique Values based on the Mean of MmColumn Unique Values using .nlargest 4F cyl. value_counts().nlargest ath. ceil(4fpg. value_counts().mean())).head Create a New Column with simple logic Useful technique for simple operations ‘sing astype(str) T can treat the floats df{'mpg") column as a string and merge it with other strings af2 = a .cony0) ‘of2{'mpg_str’] = aF2[‘nane"] + * Has HG * + AF2["RpE]-astype(str) SET efficiency = ‘poor’ WHERE mpg < 10" 2 1oe{ (4F2.mp4. < 10), “eFfLetency’} = “poor” 72 Loe{ (aF2.mpg >= 10) & (aF2.mpg < 30), “eFFictency'} ~ "nediun” F2-loe{ (A2.mp@ >» 30), “eFFickeney") = *high” arta) GroupBy and Aggregateoe ID: 5689595 Pandas makes it prety simply to group your dataframe on a value or values, and then aggregate the other results I's ait less lexibe than SQL in some ways, but stil pretty powerful. There's a lot you can do in Pandas with GroupBy objects, so definitely check jhe documentalion. 4 setting as index to False will heep the grouped values as 1 regular coluans values rather than indices arouped at = aF.groupby(by-("ey1"]) 4 use .og9 to aggregate the volues and run specified Functions # note that we con't create new columns here aggregated. nead()oc ID; 6589695 COMP3321 - A bit about Bete a eaagatps 3 00s oy ine Ses 3 omc (U) This notebook gives an overview of some basic geolocation functionality within Python. (U) A bit about geos (U) This notebook touches some ofthe random Python geolocation functional # fun this if on LABBENCH NOTE: geopandas REQUIRES running “apk add geos gdal-dev # from 0 terminal window import ipyaeps packages = ["geopy', "geopandas', "bokeh") for 4 in packages: ‘nydeps. pip(t) (U) Measuring Distance (U) Geopy can calculate geodesic sistance between two points using the Vincenty distance or greal-ccie distance formulas, witha default of Vincent available asthe class geopy.distance.distance , and the computed distance avalabe as atributes (@.,, niles, meters , ee) fron geopy. distance inport vincenty fort_neade nd = (29.10711545,-76,7460704220387) aurora_co = (59.729432,-164.6319196) print(vincenty(fort_meade_nd, aurora_co).niles, “Miles")Boe ID; 6589695 from geopy.distance inport great circle harrogate.uk = (53.9921491,1.5391039) aurora_co = (39.729492,-104.8319196) print(great_circle(narrogate_uk, aurora_co).kiloneters, “Kiloneters") (U) Getting crazy with Bokeh and Maps! (U) This information comes from ths. oreat notebook. (U) We can add map tiesto Bokeh pots to better show geolocation information! We will use some generic latlon data, found r= Anport pandas as pa us_cities = pd.Datarrane().fron_csv(us_cities.csv", Index col-None) us_efties.nead() (U) Define the WMTS Tile Source (U#FKO} Adding the ie source is as easy a defining the WMS Tie Source, and adding the tle to the the map. Note: you need your ntelink \VPN spun up to connect io this server. ‘from bokeh models inport wifSTileSource, THSTilesource (U) You also need to convert the lat and ion to plot correctly on the mercator projection map:oe ID: 6689595 ‘nport math {HeAMETHODS FOR LAT/LONG TICK FORMATTING ef prosdegToRad deg) return (deg / 189.0 * math. pi) def 1at_lon_convert(n,lat_or_lon,1s0eg-True): n= #loatin) lon = 0.8 AF Asvee: 1» prosoegToRad(n) Af Lat_or_lon ~~ “Iatitude" ‘return sma © math. 2og( (math. sin(n)/2.0)/math.cos(n)) ‘214 1at_or_ton =~ “Longstude': ‘return sm_a°(n-2one) us cities[ mere x'] ~ List(nap(lanbda x: 1at_lon_convert(x ‘longitude’ ),us_cities.ng)) vus_cities[ mere y'] » List(nap(lanbéa x; 1at_lon_convert(x, ‘latitude’ ,us_cities.lat)) (U) Finally, plot the data! ‘fron bokeh.plotting import output notebook, show fron bokeh charts inport scatter from bokeh.resources import INLINE ‘utput_notebook(resources-INLINE) P = Seatter(us cities, x='merc x", y='merc y", title="Positions of US cities") p.add_t1e(NGA MAP) fvpn ts necessary show(o) (U) GeoPandas (U) ceoPandas sa project to add support for geographic data to pandas objects. Itcurrenty implements Geoseries and GecoataFrane types hich are subclasses of pandas.series and pandas.0ataFrane respectively. GeoPandas objects can act on shapely geometry objects and perform geometne operation.oc ID: 6689695 {8} GeoPandas goometry operations are cartesian. The coordinate reference system (cs) can be stored as an attribute on an objec, and is automatically set when loading from a fle. Objects may be transformed to new coordinate systems with the v_crs() method. There is curenty no ‘enforcement of lke coordinates for operations, but that may change inthe future. ‘from geopandas import GeodataFrane from natplotlib import pyplot a6 plt ‘from shapely. geonetry.polygon import Polygon ‘from descartes. import PolygonPatch poly = Polygon({(1,3)+(342)4(1-543)4(74794(5,44(23)D) BLUE = "#6699 fig = plt.Figure() ax = Fig.gca() ax. add_patch(PolygonPatch(poly, Fe-BLUE, alpha-o.s, zonder=2)) ax.axis("scaled") pt. show)Do ID: 6589695 Module: My First Web ee Beton pase ‘months ago by[————Tin Coie dizi * 3 oo (U) Module: My First Web Application (U) A Word On Decorators (U) We've leamed that functions can acoapt functions as parameters, and functions can retum functions, Python has @ bit of special notation, called ‘decorators, that handles the situation where you want to add extra functionality to many diferent functons. I's more likely that you will need to use and understand decorators than itis that you would need to write one, but you should still understand the basics of what's going on. (U) Suppose there area several functions, al returning stings, that you want to "sign" .e. append your name to. ef doubter(to_print): return to_print*2 def tripler(to_print): ‘return to_print’3 ovber(*Hel1o!\n") (U) We an define a function that accepts this function and wraps it up with the functionality that we want: ef signer(F): def wrapper(to_print) return #(t0_print) + "\n--Mork' return wrapperoe ID: 69695 (6) Tereiterate: the argument to signer isa function, and the return value of signer is also a function tis a function that takes the same ‘arguments as the argument ¢ passed into signer, and inside wrapper , ¢ 1 called on those arguments. Thats why something lke this works: signed_doubler = signer(doubler) sigpad_tripler ~ signer(tripler) signed doubler("Hello!\n") signed tripler("Hello!\n (U) tt we are wiling to replace the original function entely, we can use the decorator syntax signer ef quadrupler(to_print): return to_print's quadrupler("Helo!\n") (U) Things get more complicated from there; in particular + function can have attributes. Therefore, @ decorator can instrument function by attaching local variables and doing something to them. + A decorator that takes arguments must generate and return a valid decorator-syle function using those arguments. + A decorator may wrap functions of unknown signature by using (*aras, **kwarts) (U) Alof this is useful when working with a complicated, multilayer system, where much ofthe work would appear tobe repetitive boilerplate. Its best o make the “business logic” (2. whatever makes this program unique) as clean and concise as possible by separating it rom the scaffolding. (U) The Flask Framework (U) Flask isa “micro-tramework’, which means that t handles mostly just the web serving-receiving and parsing HTTP requests, and sending back properly formatted responses. In contrast, a macro-framework, e.g. Django, includes its own ORM for database operation, has an integrated ‘ramework for users and authentication, and an easy-to-configure administrative backend, Because It offers So much, i takes along time to get started with Django. wew] (ven) Amport ipyaeps Apydeps.pip( “flask )Doc ID; 6589695 fron Flask inport Flask app = Flask(_nane_) app. route("/") def hell0(): return "Hello World” app.run(host="8.0.8.8" ,port=8999) # open ports:8000-9000 (U) Press to stop your app. (U) View Functions (U) A view funetion is anything for which a route has been determined, using the gapp.rovte decorator. can retum a variety of types-we've already seen a string, butt can also retum a rendered template ora Flask. Response , which we might use ifwe want to set custom headers. ‘fron Flask import request, make response, redirect, url for fruit = (‘apple :*red", "banana: yellow’ , “eranberry': crimson’, ‘dat ‘brown’ } app. route(/frutts/") ef fruit last): frutt_str = “cbr />° Join({"A 4s ()".format(") for 4 in fruit. stens())) form str = **"ebroAdd something:

“input type="text” nane="frutt_nane"» ‘input type="text” nane="fruit_color"> header str = "*"chtml> chead> boay> {0 lock body %) 1% endblock X} ‘ + Foust Asheoc ID; 6589695 (% extends “base.ntml” X) {% block body iapFruit Stande/ht> eppavailable fruit:

ule {6 for fruit An fruits. dtens() %) Lea hrot=/fruits/{{ fruit[e] }}/">U{ Frutt{e] He/a> ({E fruie{a] 4 endfor =) “pod a new Frutt

Nome
‘label fors"frutt_color"»Color click on “CA Chains" under "Server Administrators) 2 (UHFOUO) Scroll down to "Apache Certification Authority Bundles” atthe bottom and cick to expand All Trusted Partners Apache Bundles” 3. (UHFEHO) Right cick on "AllTrustedPariners.cr” and save into the directory holding your p12 certificate (U) HTTPS and PKI with requests (U) To use Pkt, you need the proper Cerificate Aulhoriy_ and PEM-encaded PKI keys. Well use @ requests. Session object so that we only have tolled these once.oe ID: 689695, Challenge: find a better algorithym than DES that dump_privatekey accepts from Openssi. import crypto p12 = crypto. load pkesi2(open("sid 0S.pi2", "rb").read{), "Your PKI password”) certfile = open("si¢_05.pen”, "we certfile.write(crypto.dump_privatekey crypto. FILETYPE PEM, pl2.get_privatekey(), ‘DES’, b°nypkipassword”)) certfile.write(crypto.dunp_certificate(crypto.FILETYPE_PEN, pi2.get_certificate())) certfile-close() tnport requests ses = rests. Sersion() ses.verify = “Apache bundle ALrustedPatners.crt LL tobe the certificate, or 0 tuple of the certificate nd password or ot least {© used to 4 tots over os geting premeel for the pesteond sesrcert = 84 D5.pe" #, bomypiporsord resp se ge Nps: ome... g00") ‘Act point you need ool oer he eminal runing your notebook ad espond othe Enter PEM pass phrase prompt. You should only get one prompt per Session(). resp. headers resp = ses.get( "https: //nbgallery.nsa.ic.gov/") resp. headers {(U) Its also easy to POST data to a web service with requests: : resp = ses.get( Andext = resp. text Find eethod poe) Andex2 = resp.text.find( "

" indext) print (resp. text[Inderl-64:Sndex2-7})oc ID: 6689695 ronoat - cnn] “subject ) Testing", “redirect "classi fication” :"UNCLASSIFTED##FOR-SrFEEEAC SEM) resp ~ s08.post(*hetps://siteworks.we.nsa.Se.gov/main/enailfore/", data-payload) print(resp.text{resp.text.find("Your form"):1) (U) Inthis exmapie, ses.cert could also be alist or tuple containing (certFile, Keyfle) . and Keyfle can be a passwordess PEM file or a PEN file and password string tuple, so you arent prompted for your password every time. (U) Low-level socket connections with socket (U) Communication over a socket requires a server (which listens) and a client (which connects) to the server, so weil need to open up two Interactive interpreters. Both the server and the cont can send and receive data. The server must 4. Bind o an IP adress and port, 2 Announce that its accepting connections, 4 Listen for connections 4. Accept a connection. ‘5. Communicate onthe established connection. Weil run the server (immediately below) inthe notebook and the client (below) in separate python window onthe system where we're running cur jupytr-notebook import socket sock server ~ socket. socket(socket.AF_INET, socket SOCK STREAM) # Pv, TCP ost = “127. PORT = 59505 # USE YOUR ow!oe 1D: 6589695 sock server. bind((HOST, PORT) ) sock server. Listen(3) sock conn, meta = sock server.accept() sock conn.send(b"Hello, welcone to the server") sock conn. reev(s095) (U) The client mast 1 Connect to an existing (IP address, pot tuple where a server is listening, 2 Communicate on the established connection ‘So or our purposes, wel run the following in a separate python window saris 15 THe CLIENT Amport socket sock client « socket socket(socket.AF_INET, socket .SCK_STREAY) Host, PORT = '227.0.0.1", 50585 # must match a hroun server sock_elient.connect((HOST, PORT)) sock_elient.recv(512) sock_elient.send(b"Thank you. I am the client) (U) Butlerng, ete. are taken care of for you, most. (U) Topics for Future Consideration: + SOAP with SOAPpy andlor SUDS. + Using modules from the Standard Library + XMLRPC + Parsing HTML witn Beautfu'SoupDec ID: 689695 HTTPS and PKI Concepts pest 9 yeas 200 YT Smo M OOO (U) Overview of HTTPS and PKI concepts, HTTPS and PKI Concepts PKI is confusing, especially given the mbx of internal and external uses, but there are some core concepts Public Key Infrastructure (PKI) Each PK certificate has two parts, the private key and the public key. The public key is simply an encrypted form ofthe private ke. tis important to oop the private key secret at al costs. A compromized private key would allow someone else to pretend tobe the original owner. Establishing Trust When you go to. amazon.com, your browser receives thir server certificate. But how do you know you can trust i? Certificate Authorities (CA's) Buried in your browser isa long ist of known certificate authorities, such as Verisign. The amazon.com server certiicate has been digitally signed by ane of these CA's. We know its coming from amazon.com because only amazon can generate the corresponding public key, and only the ‘corresponding private key can decrypt traffic sent to the public key. In other words, because your computer knows the public key is signed by ‘known CA, and your computer is sending data to that pubic key, only amazon can decrypt it because they have the corresponding private Key.oc ID: 6689695. PKI in the IC “The 1, including NSA, has its own carticate authorities (CA's). Futhermore, both the users and the servers have certificates (generally only servers have certificates onthe outside). These certificates are signed by the IC CA's, which are visible at isk nab sac. covinapesicerifialeAuthories shim Digital Signature (DS) Certificate 90% of the tie, you're using your digital signature certificate, This certificate verifies that you are you to the various services you access on [NSAnet. You also use your DS certificate for Secure Shell (SSH) to access systems lke MACHINESHOP, LABBENCH, and OpenShit. Key Encryption (KE) Certificate (nthe rare occasion that you encrypt an ema, you use your KE certificate. Your browser doesnt actually need his certificate Key Formats PKCS12 NSA keys come in PKCS12 (p12) format. contains both the public and private key. With Python, you need the OpenSSL package to use PKCS12 cericates, PEM PEN formats by far the most widely supported format on the outside. Many languages and framworks only support PEM, not PKCS12. However you can convert your key from PKCS12 to PEM formal using the openssl command, To futher complicate matters, many languages and frameworks only support unenerypted PEM certificates. You can unencrypt your PEM or PKCS12 certificate with the openss! command, but this is generally a no-no since it would allow anyone to masquerade as YoU. PPK PPK format is only used by PUTTY, the SSH tool for Windows. You can convert your key from PKCS12 to PPK format with the P12_to_ PPK Converter tooee I: os3595 PKI with Python pypki2 Examples at nso co sac govlovnanynkhiobmasiae README md By Hand with ss! Package ‘SL isthe Secure Sockets Layer, which implements HTTPS (ypar Text Transer Protocol Secure) Python 2.7.9+ ‘from getpass import getpass ‘fron rllib2 import build opener, HITPCookieProcessor, HITPError, MTTPSHandler, Request Anport ssi penbassud = gotpassnd( “Enter your PKI passwor context » s51.SSLContext(ss1. PROTOCOL sSLv23) context, Ioad_cert_chain(penCertFile, KeyFile-penteyFile, password:penPasswd) context. oad_verify_ locations (cafile-penCAFile) ‘opener = butld_opener(HTTPCookieProcessor(), HTTPSHandler (context-context)) req = Request “https: //udkipedia.nsa.ic.gov/en/Colossally.abundant_nusber') resp = opener.open(rea) print (resp.read()) ) Python 3.4+Bo ID: 6589695 ‘from getpass import getpass ‘from urllib.request import build_opener, HITPCookieProcessor, HITPSHandler, Request import ssi penBassud ~ getpassud( "Enter your PRE passvor context = 551.SSLContext(ss1_PROTOCOL_SStv23) context. load cert _chain(pesCertFile, KeyFile:penfeyFile, password-penPasswd) context. load_verify_ locations (cafile-penCAFile) ‘opener = butld_openar(HTTPCookieProcessor(), HITPSHandler (context=context)) req = Request(“https://udkipedia.nsa.ic.gov/en/Colossslly,abundant_nusber') resp » opener.open(rea) prine(str(resp.read{), encoding=‘utf-8')) read() returns bytes type, which has to be converted to str type External Packages OpenSSL ‘Handles PKCS12 and many other key formats, but not part of the standard library. Itis included with Anaconda/Jupyter. Requests ‘Supports only unencrypted PEM format. Takes care of a lot of litte things for you tke HTTP redirects. More on HTTP Status Codes ‘athiissiwikinedia nsa ic govlenList of HTTP status codesoe ID: 6589695, Updated 3 months ago by[________Jin GOMP 3524. ae (U#F@UO) This notebook demonstrates how to interact with web resources over HTTPS when using LABBENCH. It primarily ‘ses the requests_pki module. UNCLASSIFIED//FOR-GFFIGIALUSE-ONLY (U#FEYO} Python, HTTPS, and LABBENCH (U#FOUO) This notebook demonstrates how to interact with web resources over HTTPS when using LABBENCH, Ituses the requests_pki and rest_api modules, (U) HTTP with requests_pki (Ter are complicated ways of tracing wth he network using bitin rare suchas ul rib, and tpl For base (secured) Intracon we can use requests. However, th Socuro Tho No, almost evening is now Pk enabled (U) Lucy ter is a module for ta! LABBENCH has nave suport or quests wich makes tan ea bar fous hips install requestsoe ID: 6589605 Amport pydeps nodules = [requests-pki', pypac’} spydeps.pip(nodules) (U) Example 1: Obligatory example of requests nport requests 1 One of the few things not yet requiring @ certificate for Secure The Net resp = requests get(http://airs.s2.org.ns9-S¢.gov/") print("Response code: {)* forsat(resp. status_code)) print("Length of content: {)° format(len(resp content))) print("Length of text: {)*-format(len(resp.text))) (U) Thats interesting. Are content and text the samo? cesp.content == resp.text (U) tums out that content stores the bytes of the response and text stores the unicode of the response. Let's look at the txt: print(resp. text) (U) That's great we want the raw HTML...which in many cases we may. However, we can render the HTML response natively within Jupytert fron 1Python.display import display, HTML isplay (HIME (resp.text)) (U) Notice that we dirt get any ofthe images that go along with this webpage, but for our purposes now this is sufficient. (U) requests_pki (U#FOYO} LABBENCH has made interacting with secure webpages trivial! Thats because the requests_pki module works seamlessly wih LABBENCH to pass your PKI with your request. Let's see how easy tis! (U) Example 2: nbGalleryoc ID: 6689695. Amport requests_pki S055 = requests. pki.session() resp = sess. get(*https://nbgallery nsa.t¢.gov/") resp. headers ‘splay(iTAt (resp. text)) (U) So maybe Jupyte isnt meant to be a fulledged web-browser after al (U) Example 3: Notebook Gallery search (U) Search the Notebook Gallry fr aterm, got the results back as JSON, and parse the JSON, This adds a new headers argument tothe GET request. (U#FEUO} Normally @ web server wil respond wih some default ype of output. That may be spplication/hent , appicetton/xml , oF something ‘ese. you dort ke that, you can ty persuading the server to give you something else using an Accept header. That wl tll the server your preferred response formate. the format you prefer to accept). Serves often support mute formats, but not al of them. beautifulsoup" “https; //nbgallery.nsa.ic.gov/notebooks™ parans = { "" : Search_term, ‘sort’ : ‘score’ ) headers = { ‘Accept’ : ‘application/json’ } resp = sess.get(url, parans-parans, headers-headers) resp.url rine (resp. text) # json.Loods() will porse a ISON string into Lists and hashes ‘resp_parsed = json. leads(resp.text) ‘ype(resp_parsed) 4 take 0 Look at tt and find what you want resp_parsedoe ID: 689695 # print the titles of all notebooks that matched your search term [ record['titie"] for record in resp parsed } (U) Example 4: Using a proxy (U) Sometimes you noed a proxy setup, particularly when working wit second party sites. requests_pkl ind pypac make this setup quite easy! sess = requests pel SessTONh parans = { ‘type’ : ‘Comunity’ ‘espFronCsE = sess.get(url, parans-parans) + ‘project’ : “ALL , ‘service’ : ‘All" } splay HTML (respFroncse.text)) (U) Example 5: Post with JSON Sometimes yout need to ‘post’ data rather than do a ‘get request. The ‘post works similar to the ‘get, but youl need to specity parameters for the ost and usually need to set the headers as wel. This one posts the parameters as a JSON object; another common content type is application/x-nw-Fore-url-encoded , in which youll need to use the urllsb brary to URL encode your parameters prior to posting them. base_url = “hetps://naningstuff-mestern.apps.0s08. platform. cloud.nsa.t¢.fov/* # with this post, we're telling the host that we are sending json, ond want to receive son # the post paraneters are sent in the ‘data’ hey, and must be json in this cose status code = © tries =o whtde not status code == 200% resp = sess.post( base_url + "GetRecord/languages/languages' , hheaders=("Accept": ‘application/json', ‘Content-Type: ‘application/ json"), data-json.dumps({" language’: ("$ne": "English')}) > status_code = resp. status code tries 1 Af tries > 3: breakee ID: 689695 Drint(resp. status code) languages = json. loads(resp. text) print (Len Languages) print languagest0]) (U) rest_api ‘The rest_api ibrar is another resource for accessing HTTPS pages on NSANet. Like requests pki, rest_spi takes care of al the PKI authentication for you, but this library is built to enable you to create whats called an'API wrapper’, which means that were wrapping our own class ‘around the API, which is designed to ust make it easir to query the API and interpret the results. API, by the way, stands for Application Programming Interface, and is basically a clearly defined set of methods for communication with a given service, or rules for interacting with data housed in a web service In genera i you want to hit a single web page, requests_pki is generally preferred because there's less overhead (you don't have to create a ‘whole cass to doi). But f you want to hit mutiple pages at a website or API, then rest_api 's probably the better way to go, (U) Example 6: rest_api with TESTFLIGHT ‘This example shows a simple class that inherits from rest_opt.AbstractiestaPr , and allows us to hit a couple of pages (called ‘endpoints! ofthe ‘TESTFLIGHT API. Notice we set host and headers as class variables. With these set, we don't have to define them every time we make a query toa TESTFLIGHT page. For each page we just add the actual page or endpoint andthe cass fils inthe rest ofthe URL. ‘pydeps.pip(rest-api") import rest_api class Testflight(rest_api AbstractRest#PT). host = ‘https: //tf-saw.testFLight.prof.nsa.ic.gov" headers = ('Accept’: ‘application/ json") def sources (self): "Returns 2 List of all sources that feed Testflight” endpoint = */SolanaService/rest/eport/sources" return self._get endpoint). 3son() ef search(self, **hvares) "Returns report summaries that match the given Keyword arguments” endpoint = "/SolanoService/rest/report/search/" return self. _post(endpoint, dataskwargs).json()Doe ID: 6689695 ‘from pprint import pprint tf = Testflight() print(tF.sourees()[:3]) Pprint(tf.search(originator='NsA', Flelds-"subject serial nipf", starts0, rowse3, sort="Newest’)) (U) Other resources + (UHFOYO) Other notebooks on the Notebook Gallery that use requests (can you modiy example 4 above to find then?) + (U#FOYO) oynk.2, an open source module for working with your P12 certificate that originated at NSA. It's not part of Anaconda and works Destin Jupyter on LABBENCH. Itworks with urliib requests instead (U) One more comment. Be careful wien you ty to display the HTML from webpages..some webpages may alec things more than you want. resp = sess. get(*https://home.web.nsa.4c.g0v/") ) and cell () at o tine, outputting the information to the screen as csv # and to the output file in pipe("|") detimited formats for table in tables: 1 for tr dn table.findall tr"): teed 3-8 For td in tr.Findan(‘td"): ‘rint("{},"-Format(td. text) end-"*) ‘outfile. write(“elenent():()|".format(j,td.text)) Sed outfile.weite("\n") print () ‘outfile-close()Doe I: (G) Notice how we can display a hyperlink to our output - this might be handy if you don't want to go to Jupyter Home to display the file. display itm (‘*.format(?[—Jabie. txt") (U) Now lets try something a little trickier. Let's pull down th homepage and redisplay the "Current Activities" buHeted li: notebook. fe are going to use the rest_opt module here. This ts a NSA specific package and has # HTTPS support boked in. Tt makes pulling wedpoges using your PKI # the package was really designed to access RESTFUL websericassond! uristring = paraneters headers = ( ‘textyhtal”, ‘application/shent xm", “application/xal;q-0.9", create an apt object for our host server api = rest_apt.abstractRestaPt (host-uristring) ‘Het the homepage from the server. If you wanted sub-pages off the server you would put 4 thot path in the querystring as something Like “/folder/page. heal”. resp = apt. get (querystring) except: print("Well that didn't work!") creote our Beautifulsoup object bsObj = Beautifulsoup(resp.text, “html. parser")oc ID: 6689695 ‘ise the .find method to get the of the HTML document. We will drCLt down to our 4 List from there. .find() only returns the first matching HTML tag, which ts OK in this # case because you *should* only have one tag in the document body = bs0bj.#ind( body") (U#FOEUE} Now for the sticky bit. From the the Chrome brower Tools-> Developer Tools console (could have done this in Firefox as well from Tools->Web Developer->Toggle Tools) | ascertained that the path through the HTML to the bulleted list | care about is section? > diviter-containeritem-container?item- containers tem147087 > divtem-contntitem-content2 > dv > div more succinctly, as xpath it is IM@id-"section2"yivi2Vav tI ividv! but BeautifulSoup does not accept xpath (whomp, whomp). If you like to use xpath the Ixml module does a decent job of parsing HTML and does accept xpath syntax. ‘ow I progress through the body object using the find next method to get to the bulleted List activities ~ body. Find next(*div',(id':*section2"}).Find.next( ‘div’ {"elass': 'feedDisplay")) (U) Now we have the right element in the activities object. We can use the str() method to get the raw HTML from the object and either print it inline in the notebook or we can just print the text using the .text attribute. # printcactivities) display HTML (activities. str_())) An easier way: using ‘select’eID: 6689695 Inthe ‘inspector view in your Developer Tools, you can right-cck on your desied tag and choose ‘Copy Unique Selector’ to copy the CSS selector path for your tag. Then you can use soup.select oF soup.select_one to navigate dretly to that tag, rather than crawing through the entre Inierarchy to get to it (Note: Iran this in Firefox, not sure what the right-click menu is like in Chrome) selector = “.rssintries > Linth-ehild(1) > aivintn-ehil6(a)" # ot Least for our version of bsé, you have to replace # nth-child with nth-of-type selector = selector.replace(“nth-child", *nth-of-type") 1 bs0bj.seLect would find al tags with thot path sj. select_one(selector)oc ID: 6689695 Module: Operations with Compression and Archives [porn oe] Ups pou 2 your ge [Ts (U) Module: Operations with Compression and Archives49695 user_steing rane usernane, city, state,7ip_code,prinary workstation json string [(*author': "Jane Austen", “title”: “Pride and Prejudice"), ("author “Fyodor Dostoevsky", “title": “Crime and Punishment”), { with open(“user file.esv","w") as Ft F write user string)Dee ID: SAE open ‘user file. json", fuwrite(Json_string) zipfile nport peste with zipfile.2ipFile(‘user_file.zip', mod af. write ‘user File.csv") 1) as 26 2 = ripfile.ziprile(‘user File.2ip") # with 2 Filename ) # with @ File on FiLe-Like object af nomelist() 2f2.naelist() 2 = ef. f4lelist[o) 2.fllenane, 2.File size [(z.filenane, 2.File size) for 2 in 2F.filelist] 2f getinfo("user #lle.esv") user_file_csv = 2f.open("user_file.esv",'n") # returns a file-Like object! fron esv import Dictteader user_data = [_ for _ in ictReader(user_File_cs¥)} print (len(user_data)) user_data(o) user_file_esv.read() user File csv. close) af extract(af.filelist{0], ‘2Fextract’)oc ID: 699695 gzip eh with gztp.open( ‘user File.csv.ge", ‘wt") as gf ef.urite("This string will be stored as text") azip_users = gzip-open(‘user_file.csv.gr') # takes o file none, returns a ftle-Like object! X= Brip_users.readlines(), azip_users. close() x03) ‘ezip_users = gzip.open("user_file.csv.gz", ‘rt") [E.user dicts - List OictReader(grip_users)) Ruuser diets (:2] with open("user_flle.cey.ge',‘rb) as F Still gzipped = f.read() still_geippedt 102] ‘fron so import Stringio unpacked users = gzip.GzipFile(Fileobj-io.stringlo(still_gripped)) # what iF you have bytes or o ftle-Like obejct co unpack? ‘unpacked users. readlines(){:3) tarfile snort tare veith tarfile.open("userFil ‘ef-add(“user_File.csv") ‘ef add(“user_File. json") W's moden'w") as tf: tarfile is tarfile(‘userfile.tar’), tarfile is tarfile(‘user file.csv") {tf = tarfile open(‘userfile.tar") # don’t need to unzip First! ‘F.petmenbers() F-getnanes()eID: 6689695 u = tfeextractfile( user file.csv") uz » thextractFile(tf..getmenbers()[1]) sresdline() 2.read()(2 50] ‘theextractal1('from_tarball')Doc ID: 6689655 Module: Regular Expressions ee ‘months ago by[ ie oD (U) Module: Regular Expressions (U) Regular Expressions (Regex) (U) Now You've Got Two Problems... ‘Some people, when confronted with a problem, think "I know, Il use regular expressions.” Now they have two problems. Jamie Zawineki, 1997 (U) A regular expression is too! or finding and capturing patterns in text strings. Its very powerful and can be very complicated; the second problem referred to inthe quote isa commentary on how regular expressions are essentially a separate programming language, AS @ rule of thumb, Use the in operator or string methods tke #ind or startssth if they are suitable forthe task. When things get more complicated, use regular ‘expressions, but try to use thom sparingly, Ikea Seasoning. AL times it may be tempting to write one giant, powerul, super regular expression, but that is probably not the best thing to do. (U) The power of regular expressions is found in the special characters. Some, ike “and $ , are roughly equivalent to string methods startswith and endswith , while others are more flexible, especially . and * , which allow flexible matching (U) Getting Stuff Done without RegexBo ID; 6589695 nike" An “so many mikes!” onike™.startewith("at") lendswith( ko") #ind(-¥") “aike" Asalpha() nike” ssdigit() mike® replace("k", "e") (U) Regular expressions in Python ‘There ae only a few common methods fr using the re module, but they dont always do what you would fst expect. Some functional is exposed through lags, which are actually constants (Le. int defined forthe re module), which means that they can be combined by addition. import re reamaten("e", “abedef") re.maten("2", “abedef") re.seareh("e", “abedef") re.searen("", “abedef") re.search("C*, “abcdef", re.t) # re. IGWRECASE re.search("*c", “ab\ncdef") re.search("*e", "ab\ncdef”,re.M) # re. MULTILINE re.search("*C", “ab\ncdef", eM + re.t) (U)Inboth natch and search , the regular expression precedes the string to search. The diference between the two functions is that match works ony at he beginning ofthe sting, while search examines the whole string.Boe ID: 6559698 {0} When repeatedly using the same regular expression, compiling it can speed up processing. After a compiled regular expression is ‘created, Find, seareh , and other methods can be called on it, and given ony the search sting asa single argument. cre = re.compite("e") €.re.search("*abede") Regex Operators = matches a single character matches beginning of a string or newline $ matches end of string = @ oF nore of something += 1 or more of something 2 = 0 or 1 of something "2, 42, 22 ~ don't be greedy (see exanple below) {3} - match 3 of something (2,4) = match 2 to 4 of something \ © escape character [rntR] = match any ONE of the letters 1, ry my Ly RN [a-n] - match any OWE of letters fron a to m [ln] = match letter a or We match a letter As = match 2 space Ad = maten a digit re.search(*f'ss", "hike Likes cheese\nand Mike Likes bees") 3 anda CGH MDERIH(2)-[H(4)", “vetto, 1 an a very bad terrorist. 1 you uanted to know, ay phone number 4s (303) 555-2 2 findali(*mi.tke", "1 am looking for mike and not all this stuff in between mike") re.findali("mi.*?ke", "A am looking for mike and not all this stuff in between mike") Capture Groups Put what you want to pull out ofthe strings in parentheses ()Dowd: 95 ww.string © tien 4s the best tess Yor dein. fesult = re.findali(™ je), my-strng) print(restit) Prine(result)) ro" gramming” Matches and Groups (U) The return value from a successful call of match oF search is a match objec, an unsuccessful call retums None. First this issuable for use in 4 statements, such as iF <_re.search("abeae”): ... . For complicated regular expressions, the match object has all the deals about the substring that was matched, as well as any captured groups, i. regions surrounded by parentheses i the regular expression. These are available via the group and groups methods. Group 0's always the whole matching string, after which remaining groups (which can be nested) ae ordered according to the opening parenthesis. = resmatch(r"(\we) (\w#)", "saae Newton, physicist") -group() groups) sn-group(2) m-groups() Other Methods (U) Other regular expression methods work through all matches inthe sting, although what i returned isnot always straightforward, especially ‘when captured groups are involved. We demonstrate out some basic uses without captured groups. When doing more complicated things, please ‘omember: be careful, read the documentation, and do experiments fo test! ve. findall ("2c beadeaeeafe") # returns List of strings re.finditer("a.c","abcadeaeeafe") # returns ‘erator of match objects re.split(ta.", “abeadeaecafe) # returns List of strings (U) The suv method returns a modified copy ofthe target string. The fist argument is the regular expression to match, the second argument is ‘what o replace it with-which can be another string or a function, andthe thie argument isthe string on which the substitutions are to be caried ‘ut the sub method is passed a function, the function should take a single match object as an argument and return string. For some cases, ifDoc ID: 5689658 the substituton needs to reference captured groups from the regular expression, it can do so using the syntax \gcnunber>, which is the same as ‘accessing the groups: method within a function Pet, "ance", “abracadabra*) re-swecra(.t2)e", “ager ecie", “abracadabra) ef reverse_First_group(natchob3): natch = matchobs.eroup() rrev_group = matchobj.group(1){::-2] ‘return natch[:matchobj.start(1)] + rev_group + natch{atchobj.end(1):) re. sub("a )c* yreverse_First_group,“sbracadabra’ (U) Inthe above, we used start and end, which are methods on a match object hat take a single numeric argument-the group number-and ‘etum the starting and ending indices inthe string ofthe captured group. {(U) One final waning a group can be captured more than once, for instance when its defniton Is folowed by a + or a * then only the last ‘occurence ofthe group willbe captured and stored.‘Doc ID: 6589695 Hashes pats mont gs fT ess sates =a (U) Computing Hashes in Python (U) Hashes (U) Lets start with hashes, Hashes map data of arbitrary size to data of fixed size and have a variety of uses: + securely storing passwords + verifying fle integrity + efcienty determining if data isthe same (U) There are many different hashing algorithms. You've probably heard of some ofthe more common ones, such as MDS, SHAT, and SHA256. (U) Hashes have some useful features: + they are one-way, meaning that given a hash, there isnt a function to convert it back tothe original data + they map data to a fixed output, which is useful wien comparing large amounts of data (such as fles) (U) Sollet’s generate a hash. ron hashlib import sha2se sha2S6("abe" encode(“ascit')) hexdigest() (Wor ‘shazs6(b‘abc") hexatgest()Doe ID: 689695 (6) We a1 know storing plaintext passwords is bad. A common technique of avoiing this isto store the hash ofthe password, then check ithe hhashes match. So we can create a short function to check ifthe lyped password matches the stored hash: ‘def check_password(clear password, password_hash): return sha256(clear password) hexdigest() =~ password hash (U) Does anyone know why storing the hash ofa password is bad? (U) Ifthe password hash database was ever compromised, it would be vuinerable to a pre-computation attack (rainbow table), where an attacker pre-computes hashes of common passwords. There are tools such as scrypt to help mitigate this vulnerabilty. (U) How about a safer use of hashes? Suppose you need to look for duplicate files? Doing a byte-per-byte comparison of every fe to every other fle would be realy expensive. A better approach is to compute the hash of eac fle, then compare the hashes. mort 0s ‘from hashlib import mds ef get_File_checksun(Filenane) h = mas) chunk size = £198 With open(Filenane, ‘rb") as wide True: ‘chunk = £.read(chunk_si26) Af Jen(chunk) == 0: beak hupdate(chunk) return h.hexdigest() (U) There is a small danger with this approach colisions. Since we're mapping alot of data toa smaller amount of data, there is the possibilty that ‘wo fles wil map to the same hash, For SHAZ56, the chances that two fies have the same hash are 1 in 2256, or about 1 in 1.168*77. So even with aot of file, the chance of a olision is small (U) Notice that we dont need to ead inthe entire fle at once. One really cool feature of hashes is they can be updated: h = shaase(b'abe) h.update(b'def") huhexdigest<) sha2S6(b'abedet")-hexdigest()oc ID: 6689695 Module: SQL and yan. stg re ant Ti oe aes ooo (U) Module: SAL. and Python (U) The Odd Couple: Programming and Databases (U) makes lt of sense to Keep your data in a database, and programming logic ina program. Therefore, i's worth overcoming the fundamental impedance mismatch between the two technologies. In the most common use cases, where the program ist too teribly complicated and the data Isnt to crazily interconnected, things usually work just fin. (U) Python has a recommended Dalabase AP although there are sight variations in the way this APL i implemented, whichis one reason to use a ‘metalbrary tke SQLAIchemy (well get to tis later). The standard brary only provides an implementation for SQLite, in the sqlite? package. Connections to other database types require external packages, such as soi (confusingly, to get this you have to pip install ySQl~ python ) Brroboy drop tables (U) Basics with sqlite3 To interact witha database, a program must 4. Establish @ connection 2 Create a cursor 3. Execute commands ‘> Read the results ‘© Commit the changesoe ID: 6589695 "2 Ghose the cursor andlor connection (U) Using a basic adapter, commands are executed by passing strings containing SOL commands as arguments, import sqlites com = sqlite3.connect(‘test.db') # SQLite specific: creates ab {f necessary cur = conn. cursor() cur-execute("*"create table fruit ( 4d integer primary Key, fname text not null, color text default "RED" insert into fruit (ane) values ("apple")"*") # not there yet conn. comit() # to moke sure it's written ‘cur-execute(""* select * fron fruit “**) # returns the cursor--no need to capture it. ‘cur-Fetchone() (U) When making changes to the database, i's best to use parameter substitution instead of sting subtituton to automaticaly protect against Lnsantized input. The sqlite3 module uses as its substitution placeholder, but ths difers between database modules (whichis a major headache when writing code that might have toto connect to more than one type of database), fruit data = [(banana’ "yellow, (Ceranberry’"erimson"), Cdate’, "brown", (Ceggplant’, "purple’), (fie, “orange’), (Carape’, ‘purple")) for £ An fruit data: ccur.execute("""insert into fruit (name, color) values (2,2) bi curcexecute(“**select * from fruit""") # DANGER! OATA HASW°T BEEN WRITTEN YET! cur. Fetenone()Doc ID: 689695 cur. fetchmany(2) cur. fotehal() (U) A cursor is itrable: more fruit = [(*honeydew', ‘green'), ("ice cream bean’, "brown"),("Jusube’, ‘red")) cur oxecutomany(""" insert into fruit (name, color) values (2,2)"** snore fruit) cur-execute("**select * from fruit”**) [sten[2) for Stem dn cur] # rood the none ccurcexecute( "PRAGHA table_info(Frust)") for Line sn cur: print Line) cur. Fetenal1() com. comit() # olways remember to commit! (U) In sqlites, many of the methods associated wth a cursor have shortcuts atthe level ofa. connection ~behind the scenes, the module creates, ‘a temporary cursor to perform the operations. We wil not cover it because it st portable (U) Other Drivers (U) The most common databases are MySQL. and Postgres. Instaling the packages to interac with them i often frustrating, because they have rnon-Python dependencies. Even worse, the most curent version of sysql-python in PYPIis broken, so we request a diferent version: vew) (vew) (WE) [16 vip instath papcepe2 «+ Error: pg_config executable not found {(U) With enough exceptions to make fe very frustrating, they work ke sqliteDoc ID: 689695 (U) SQLAIchemy (U) SQLAIchemy is very powerful, very complicated package that provides abstraction layers over interaction with SQL databases. It includes all kinds of useful fetures lke connection pooling. Well discuss two basic use cases; in bot of which we just want to use ito get data in and out of Python. (U) Cross-Database SQL {U) Imagine the folowing scenario: during development you's lke to use SQLite, even though your production database is MySQL. You dont plan todo anything fancy: you already know the SOL. statements you want to execute (although there area couple of things you always Wished sqlite would do for you, ke retuming a dict instead of @ tuple (U) Enter SQLAlchemy. does require that you have a driver installed, e.g. MySQlLdb, to actully tak tothe database, but it takes care ofall the tickytack syntax details. By default, it even commits changes automaticaly! import tpydeps spydeps.pip("salalcheny") Amport salalcheny engine = sqlalcheny-create_engine(: ite:///Rest.db") # database protocol and URL result = engine.execute(‘select * fron fruit") ans = result. fetchall() first ans © ans(01 *ypecinst_ans) frst_ans(2] fipst_ans. keys) *trst_ans.values() engine-enceute("' ‘insert Anto frust (name) values (2)°°*,Ckunqust’)) engine.execute(‘““Snsert Snto fruit (name,color) values (?, 2)'"",{(* Line’, ‘green’ (mango, "ereen")1) result = engine.execute("select * fron fruit’)oe ID: 6689695 result. fetehal() (U) Now, to move to MySAL, all you have todo is use a diferent URL, which folows the patter: ‘ialectsdriver://usernane:passuordmost:port/ database The SOLAlchemy documentation lists al the dabases and crivars (U) As Object Relational Mapper (U) The real power in SQLALchemy i in using it to store and retrieve Python objects from a database without ever writing a single line of SQL. It takes alte bit of what looks lke voodoo at frst. Well skip most ofthe details for now, a the risk ofthis being a complete cargo cult activity. Open pa new fe called sql_fruit.py and put the following into it: ‘from sqlalcheny inport create_engine, Column, Integer, String, Date fron sqlalcheny.ext.declarative inport declarative bare fron sqlalcheny.orn inport sessionaker engine = create _engine(*sglite:///test.db*) Base = declarative base() Session = sessionaaker(bindzengine) session = Session() lass Frust(sase): \_tablenane__= “fruit Ha = colum(integer, primary key=True) rnane = Colunn(string) colorsColuan(String, default="Reo") ef _inkt_(self, name, color): Felf-nane = nave self.color = color ef _repr_{(sel#): return ""-format(self.id, self.name, self.color) (U) Now, in the interactive interpreter: ‘from sql fruit import * Fqvery = abs son. query Fruit)99695 query. 013() query. #iest() nectarine = Fruit(’nectarine®, ‘orangered") session. ada(nectarine) o_sesston.comit()oe 1D: 6689698 Easy Databases with sqlite3 a coe os 3 years ao by Jin COMP 3524 (U) Example on using sqiite3 to group and average data instead of using dictionaries. Easy Databases with sqlite3 ‘The great thing about sqites i that it allows you to create a simple, local database without having fo install any servers or other tools. The entire ” + aml_records + “* saith open("file.xal, “w") as Fawrite(xml_records) (U) Ina rapidly. developed prototype with controlled inp, tis may not cause a problem. Given the way the real world works, though, someday this litle snippet from a one-off script will become the long-forgoten key component ofa huge, enterprise-wide project. Somebody wil ry to feed it data in just the wrong way ata crucial moment, and it wl fal catastrophicaly (U) When that happens, youl wish you had used afully-developed library that would have had a beter chance against the malformed data ‘Thankful, there are several~and they actually arent any harder to get started with. (U) Comma Separated Values (CSV) (U) The most exciting things about the csv module are the OsctReader and Dicthriter classes, Fist, ls look atthe pain vanila options for reading and writing, Amport csv # = open("user_file.csv") reader = esv.reader(#) header = next (reader) ali_Lines = [Line for Line in reader al_Aines. sore()oe ID: 6589695, 8 = open('user_File_sorted.csv’ weiter = cav.wrster(g) setter writeron(header) veiter writerows(al1_tines) B-close() (U) CSV readers and witers have other option involving dialects and separators. Note thatthe argument to csv.reader: must bean open fle (or fle-ike object), and the reading stars at the current cursor position (U) Accessing categorical data positonaly isnot deal. That is why csv also provides the octReader and oicturster classes, which can also handle records with more or les in them than you expect. When given ony a file as an argument, a DictResder uses the fst line as the keys for the remaining lines; however its also possibe to pass in Feldnanes as an adltonal parameter F s00k(0) a.reader - csv. bictReader(#) records = [ine for Line in d reader} (U) To see the diferences between reader and DictReader, look at how we might extract cies from the records in each, 4 for the object fram esv.reader citiese = [record[2] for record in all_lines} 1 for the object from csv. DictReader ceittest = (record{ ‘city"] for record in records] cities == cities (U)Ina oiceiriter, the ¢ieLénanes parameter is required and headers are not writen by default If you want one, add it with the writeheader method. Ifthe ¢ieldnanes argument does not include all the fields for every dictionary passed into the Dictwriter, the keyword argument extrasaction must be specified. 1 = open(*nanes only csv", “x") d.uriter = csv.Dicthntter(g, [‘nane", ‘primary workstation}, extrasaction-"ignore") 4 weiter wrsteheader()oe HD: 6589695 writer weiterows (records) g-close() (U) Javascript Object Notation (JSON) (U) JSON is another structured data forma. In many cases itlooks very similar to nested Python dict sand ist s. However, there are enough Notable differences from those (e.g. ony single quotation marks are alowed, boolean values have a lowercase inital letter) thats wise to use & dedicated module to parse JSON data Sil, serializing and deserialzing JSON daa structures is relatively painless. (U) For this section, our example willbe a ist of novels: Amport json novel ist = (1 novel List. append({"title": "Pride and Prejudic author’: °3 austen") novel_1ist.append({ "title" ‘Crime and Punishment’, ‘author: ‘Fyodor Dostoevsky"}) novelist. append( {tit + "The Unconsoled', ‘author’: ‘Kazuo Ishigure'}) Json.dumps(novel List) # to string eth open( ‘novel List. son’, 'w") as f: Json.dump(novel_list,f)” # to file the hobbit = "{"title": "The Hobbit", "author": °3.R.R. Tolkien*}’ rnovel_1ist.append(json.loads(the hobbit) # frow string saith open("war_and_peace.json") asf: # <-- if this file existed novel List. append json. load()) * from Fite (U) By defaut the 1084 and 1osds methods return Unicode strings. Its possible to use the json module to define custom encoders and decoders, but this isnot usually required, (U) Extensible Markup Language (XML)oe ID: 689608 {(G)'This lesson is supposed tobe simple, but XML is complicated. Well cover only the basics of reading data from and wing data to fies in a very ‘basic XML format using the ElementTreeAPI, which s just the most recent of at leas three approaches to dealing with XML inthe Python Standard Library, We wil not discuss attrbutes or namespaces at al, which are very common features of XML. f you need to process lots of XML quicly I's probably best to look outside the standard lary (probably at a package called Lami). (U) Attrough there are other ways to get started, an Elenenttee can be created from a fle by italizing with the keyword argument File ‘fron va.etree inport Element Tree ml_file + open(‘user_file.xnl") user_tree = ElementTree. ElenentTree(fie-xal_ fie) (U) Todo much of anything its bos 0 pull he rot element out of the ElenentTree . Elements ar iterable, so they can be expanded in lst Comprehensions. To see what s inside an element, the Element Tree module provides two class functions: dum (which prints to screen and retums one ) and tostring . Each node has a text propery, although in our example these ae al empty except for leaf nodes. root elt = user_tree.getroct() sere ~ [u for u An root. elt] bring Elenenttree. tostring(users(®])) wehildren = (x for x An users{9]] wehtldren(2}.text sw ehildren[2].text = ‘north-x5-1234" tenant Tree. dump(users{01) (U) Toget nested descendant elements directly, use ¢indall, which returns alist of all matches, or Find, which retums the frst matched element. Note that these are the actual elements, not copies, so changes made here ae visible inthe whole element tree. all_usemanes = root_elt.findall("user/nane/usernane’ ) [n-text for n in al1_usernanes{:10}] (U) To construct an XML document: + make an Elenent ‘+ append other Eienent s tot (epeating as necessary). + wrap itall up inan elenentiree , and + use the Elenenttree.write method (which takes a fle name, nota File object)oe ID: 6689695 apple = ElementTree.Lonent( apple") apple.attribf color") = ‘red apple. set("varsety', “honeyersp") apple-text = “Tasty” Elenent Tree. dump(apple) ‘Frult_basket ~ lenentTree. Elenent( basket!) ‘Frutt_basket.append(apple) Frutt_basket append(ElenentTree.2L("")) Elenentree, dunp(Frutt_basket) Frult_tree ~ Elenenttree. ElementTree(Frust_basket) fruit tree.write("fruit_basket wal") (U) Bonus Material: Pickles and Shelves (U) Atthe expense of compaiity with other languages, Python also provides builtin serialization and data storage capabilties inthe form of the pickle and shelve modules. (U) Pickling Snport pickie picklene = ( plcklene["Titie"] = ‘Python 48 Cool 3 pleklenePagecount®} = 523 ptextene author) - [———}oe ID: 6589695 wath open("/tmp/pickleddata.pick', ‘xb") as p: P = pickle.dunp(picklene, p) saith open("/tmp/pickleddata.pick', rb") as p: P = pickle. load(p) rint(p) (U) Shelving (U) Creating a Shelve ingore shelve pletion = () plcklene("Title"] = ‘Python Ss Cool” plckieme[‘Papecount"} = 583 ———) ‘do = shelve.open" /tmp/shelve. dat) ickdeme[ ‘Author ] ecb) «petene aaned picklene["Title’] = ‘Python 4s Cool -- The Next Phase’ pti escort) = 123 nietee'acnrs) - [——] ar teot) «pee amet) .close()Doo ID: 6 (BY Opening a Shelve ‘do = shetve.open(*/tmp/shelve cat") 2 = db.keys() 2 = db "book *] = db "book2"] print) print) print) .close() (U) Modifying a Shelve do = shelve.open("/tno/shelve.dat') keys) 2 = dof" bookt"] = do "book2") prine(a) print(o) print(a) account} = 588 eecount") = 123 ae["bo0ka"} = a a["booka"} = &oo ID: 6689695, ‘do, close() UNCLASSIFIEDDoe ID: 6589695 Module: System Interaction & oo (U) Basie operating system interaction using the os, shuti, and sys modules. fated over 3 years ago by UNCLASSIFIED (U) Introduction {(U) Pytion provides several modules for interacting with your operating system and the les and directories it holds. We wil talk about twee: os , shutil and sys (U) Be aware that while this notebook is unclassified, your output may not be (depending on the fles you're displaying). (U) os Module: (U) This module helps you interact with the operating system, providing methods fr almost anything you would want todo ata shell prompt. On POSIX systems, there are over 200 methods in the © module; we wil just cover the most common ones. Be aware thatthe o8 module includes ‘methods that are not cross-platform compatible; the cocumentatin fs helpfully annotated with Availability tags. (U) Directory discovery and transversal is prety basic: import 05 os-getoud() os.ehdin(*/tmp') # Unix dir--choose different dir for Windowsoe ID: 6689695 os. Listdir() os. getoui() walker = os walk(os.curdir) ‘ype(waiker) List(vater) (U) Avoid one common contusion: os.curéir is a module constant (°.” on Unictike systems), while os. getexs() i a function. Either one can be used in te method os.walk., which returns a generator that traverses the filesystem tree stating a the method's argument. Each successive value rom the generators @ tuple of (ainectory, [subdirectories], [files]) (U) A variety of methods allow you o examine, modiy, and create or remove directories and files. = open(‘nen_temp_file.tat, ‘w') F.close() os. stat(‘new_tenp_File.tet") s.mkain(other_dir*) s-renane( ‘new tenp_flle.txt", ‘other dir/tempfile.txt*) (U) The os.path submodule provides addtional functionality, including cross-platform compatible methods for constructing and deconstructing paths. Note that while itis possible to jin a path completely, deconstructing a path occurs one element at atime, right o left. ample path = 05-path.Join(ford, "trucks", "f250") sample_path os.path.split(sample path) 0s. path.exists(sample_path) (U) Information about the current environments also available, ether via specific methods orn the os.environ object, which functions ike a dictionary of environment variables. If os.environ Is modified, spawned subprocesses inherk the changes. os. getlogin()oc ID; 6589695 os.getutd() # unix os. getgroups() # Untx ‘os.environ{ ‘NE_TENP_VAR"] = "323456" s.uname() # Unix (U) shutil Module (U) Living on top ofthe 0s module, shutil makes highevel operations on fles and collections of les somewhat easier In partcular, functions are provided which support fle copying and removal, a well as cloning permissions and other moladaa, Amport shut shutil. copyfile(are,dest) # overurites dese shutil.copynode(sre,dest) # permission Dies shutll.copystat(sre,dest) # perstssion bits and other metadata shuttl.copy(sresdest) # works Like cp Sf dest 4s 2 dtrectory shutil.copy2(sresdest) —# copy then copystat shuts. copytree(srcy dest) shutil.rwtree(path) # must be real directory, not o symlink shuthl move(sre,dest) # works with directories (U) sys Module (U) The sys module provides access to variables and functions used or maintained by the Python interpreter, it can be thought of as a way of ‘accessing feature from the ayer between the underlying system and Python. Some ofits constants are interesting, but nt usualy usefuloe ID; 6589695 Amport sys sys.maxsize sys.byteorder sys.version (U) Omer module atributes are sometimes useful, although fiddling wth them can introduce problems with compatibility. For instance, sys.path Is alist of where Python wil look for modules when ingore is called. I itis modified within ascrpt, and then modules can be loaded from a new location, but there is no inherent guarantee tha location wil be present ona system other than your own! On the other hand, sys.exit() can be sed io shut down a script, optionally returning an error message by passing a non-zero numeric argument. UNCLASSIFIEDac ID: 689695 Manipulating Microsoft Office Documents with win32com SD ae (U) Demonstration of using win32com to create and modify Microsoft Office documents. eat (U) Manipulating Microsoft Office Documents with win32com (U) Welcome To Automation with win32com! (Te wind2e0m module connects Python othe Mcosot Component Object Model nerfae that enables inter-pocess communication and thject creation witin Micoeat Os appcatons. (U) Note: wins2com only exists on Windows platforms, so this notebook will not run on LABBENCH. In order to run this notebook, install ‘Anaconda3 on your Windows platform and use jpyter-notebook (U) "Hello World" for Word (U) We need to impor the library, and open Word. import wins2con.client word = win32con.client.Otspatch( "word. Application’)Doe ID: 69695 (G) Cispaten checks to see i Word is already open. Its, it attaches to that instance. I you's tke to always open a new instance, Use bispatcnex (U) By defaut, Word wil start, but wont be visible. Set this to Tre it you want to see the application, word. Visible = True (U) Greate a document and add some tox, setting @ font size that we like. orddoe = word. Documents Add() orddoc content. Text = "Hello Worle orddoc Content Font. Size = 18 (U) Save the document and exit the application. Note that win3zcon bypasses the normal Python fle object, so we need to account fr the Windows directory separatr. (UAFELO} Also, CiassifyTool may nag you for a classification. In order to prevent thi, in Wor under When Closing Document’, uncheck "Atways show Classifiation Form’, and click "Save". worddoc. Savess("u:\\private\ \Jupyter\ \win32com\\hel10.d0cx") word. quit() ‘select the “CiassifyToo!” tab, click on "Options", and (U) Thats it (U) More Elaborate Word Example (U) There's another option fr stating the application: word = wina2com.client.gencache. EnsureDispateh( "Word. Application”) (U) This can take slighty longer, but enables access to win32com constants, which are required for some methods. The alternative is to look ‘through the win32com documentation forthe value ofthe constants you need, (U) Lets take a look ata possible use case. Say we have reports in a particular format that we ned to regularly generate. We can create a template wit the sections that willbe replaced. In this case, they are eportevent , ReportTine , and ReportPlace . First, dovnload the tempol ‘Them open the template and create a dictionary withthe sections and the data that wll be used,oe ID; 6589695 ‘constants = win32con.client. constants 1 save some future typing word. Visible = True wordioc = word. Docunents. open( u:\\private\\jupyter\\win32com\\deno_tenplate.docx') event details = { "Reportevent™ : “ReportTine™ “neporePlace™ » (U) Now the magic happens. Lets iterate through the dictionary, replacing al ofthe sections withthe data. # Beecute( FindText, MatchCase, MatchiholeKord, MatchWiLdcards, MatehSoundsL tke, MatchALMordForms, + Forward, Wrap, Format, Replacewtth, Replace) for tag, data in event details itens(): ~ word. Selection. Find.Execute( tag, False, False, False, False, False, True, constants.wiFindContinue, False, data, constants.wdeplaceAl1) (U) We can ads a couple of paragraphs of addtional info, and we're done. paragraph = worddoc.Paragraphs.Add() paragraph Range.Text = ‘Additional info\n" footer ~ worddoc. Paragraphs. Add() Footer Range.Text = "Produced by me\n* worddoc. aveAs(u:\\private\ \Jupyter\ win32com\ \deno_out. docx") word. quit() (U)PowerPoint (U) PowerPoint works very similarly. Again, ownload the template ppt = win32com. client Dispatch('PowerPoint Application") resentation = ppt.Presentations.Open( u:\\private\\jupyter\\wins2cem\ WyTean_tenplate.pptx") (U) Did you notice that we didnt need to set ppt.visibie ? PowerPoint is always visible. title = presentation. s1ides(1) (U) We know the fst side is the ttle slide, so we've seta variable tot. PowerPoint presentations are made up of sides, which in tum are collections of shapes. To modify a presentation, we need to know which shape is which. Lets take a look atteDoe ID: 689695 title (U) Hmm. That air(titiey not very helpful Lets see what methods we have: (U) Atthis point youre probably realizing that COM objects dont act ike normal Python objects. help(titie) (U) So Python just takes anything you try to do with «tie and passes ion to the Windows COM library. Which means youll need to consult "Microson's Win32Com documentation i you have questions about something, (U) Lets get back to working with this presentation. We stl need to ind out which shape is which: for 1, shape in enunerate( title. shapes) shape. TextFrane.TextRange.Text = ‘Shape #{0)' format(1+1) (U) This sets the text for each shape to Is index number so we now have a number associated with each shape. You only need to do this when youre wrting your script. Once you create your template, the shape numbers won' change. So the tie is #1 and the subtle #2. (U) undo a fow times wil remove the numbers. (U) Lets update the tie slide with todays dat: ‘from datetine inport date today = date. today().strftine( "xViad") ‘ite. Shapes(2).TextFrane.TextRange. Text ~ today (U) Now ls update the status of our two focus areas. Weil ski the step of identiying the shapes we want to mesify Focust = presentation.s1ides(2) focust.Shapes(2).TextFrane. TextRange.Text - “ALL Good, 805s" focus? = presentation.s1ides(3) Focus2.shapes(2).TextFrane. TextRange.Text = "Sir, We have 2 probles’ (U) Now save the presentation with todays date, and Bob's your unde. presentation. Savehs("u:\\private\ \Jupyter\ win32eom\\WyTeam_(0) ppt’ format (today)) presentation. close() pot.qutt()Doe ID: 6559695 (U) Visio (U) Staring the application should look familar: visio = win32con.cltent Dispatch(*Visto.Application") ‘documents ~ visto.docunents ocunent = documents.Add("Basic Network Diagran.vst") # Start with o butlt-tn template document Title = "New Network Graph” # had 0 title pages = visto, ActiveDocunent Pages page = pages. Ttem(1) (U) Visio is visible by default, but can be hidden if desired (U) So we've created a document and grabbed the page associated wit it. Visio shapes are part of stencil packages, soles add a couple. Networkstencil = visto.Oocunents.Addtx( "periph n.vss", 9, 16464, 0) Conputerstencil = visto. ocunents.AddEx( "Computers and Honttors.vss", &, 15+6%, 0) (U)Otner stencils are + Network Locations: netloc_n.. + Network Symbols: net sym, + Detailed Network shapes: ctnet_a.vss + Legends: agn_n.vss {(U) Other stencil names can be found on the Internet. (U) Now we need the shape masters that wel use. pe = Computerstencil Masters. 1tem(“PC") router = NetworkStenctl Masters. Tten("Router") server = NetworkStencsl Masters. Tten("Server”) comector = NetnorkStencil. Masters. iten( "Dynamic Connector") (U)The names match the names you see when you view the shapes inthe stencil sidebar. Lets add a few shapes.oo ID: 6689695. ‘pct = page.orop(ec, 2, 2) pet-Text = *10.1.1.2" e2 = page.Orop(pe, 12, 10) pe2.Text = "20.2. Server ~ page.Drop(server, 15, 5) servert. Text = °10.2.1.200" router = page.Drop(router, 8, 8) routera. Text = °10.2.1.250" (U) Some of the shapes went off the page, so resize. You can walt unl the end todo this, but its more fun to watch the connections being drawn, page. ResizeToFitcontents() page. CenterOraxing() (U) Now draw the connectors. arrow = page.Drop(connector, & °) arrowBegin = arrow.Cel1sU("Seginx").GlueTo(pct.cel1su("Pinx")) arround = arrow.cellsU("EndX").GlueTo(routert.cellsu("PinX")) arrow.Text = "pel connection" (U) We can customize a connector arrow = page.Drop(connector, ©, &) arrowBegin ~ arrow. Cel1sU("BeginX") .GlueTo(pe2.Cel1sU("PinX")) arrowEnd = arrow.Cel1sU("En@X").GlueTo(routert.celisu("Pinx")) arrow.Ce11sU("LineColor"). Formula = "=RGB(255, 153, 3)" arrow.Cel1sU("Endarrow").Formula = "5" arrow.Cel1sU("EndarrowSize").Formula = "4" arrow.Cel1sU("Lineiesghe™). Formula = arrow.Text = "pe2 conection" 1 arrow = page.Drop(connector, ©, 2) arradBegin = arrow.Cel1sU(Beginx") .GlueTo(servert.cel1su("PinX")) arrowénd = arrow.Cel1su("EndX).GlueTo(routert.Cellsu("PinX")) arrow-Text = "server connection* (U) Now resize, ecenter, and save.oe ID; 6589695 page. ResizeToFitcontents() page. CenterDrawing() document. Saveks"U:\\private\ \Jupyter\ \win32com\\visio_demo.vsdx") (U) Close the application. visto.quit() (U) Excel (U) win32com works with Excel too, but due to the slowness of the interface, you're probably better off using pandas,oc ID: 6589695 Module: Threading and Subprocesses (U) Module: Threading and Subprocesses (U) Module: Threading and Subprocesses (U) Concurrence and Python's GIL - i.e. Python doesn't offer true concurrence (U) Python's Global Interpreter Lock (GIL) means that you can really only have one true thread at one time. However, Threading in Python can be immensely helpful in speeding up processing when your script can perform subsequent steps tha do not depend on the ouptut of other steps. Basically, it gives the illusion of being able to do two (or more) things at the same time. (U) Threading (U) Threading allows you to spawn off "mini programs" called threads that work independently of the main program (sort of). Threading allows you to send data off to a function and let it work on getting results while you go on with your business. ItDec ID: Can also allow you to set up functions that will process items as you add them to a work queue. This could be especially helpful if you have parts of your program that take a long time to execute but are independent of other parts of your program. A good example is using a thread to execute a slow RESTful web service query. (U) This adds some complexity to your life. Threads act asynchronously - meaning that you have limited control as to when they execute and finish. This can cause problems if you are depending on return values from threads in subsequent code. You have to think about if and how you need to wait on thread output which adds extra things to worry about in terms of accessing data. Python provides a thread- safe container named Queue. Queues will allow your threads access without becoming unstable, unlike other containers (such as dictionaries and lists) which may become corrupted or have unstable behavior if you access them via multiple threads. (U) Subprocess (U) The subprocess module is useful for spinning off programs on the local system and letting them run independently. import ipydeps nodules = ["threading’, “queve"] for m in modules: {installed packages = (package. project_nane for package in ipydeps._pip.get_installed dtstributions()] Af (n not An installed packages) and (m not in ipydeps. sys. modules) ‘pydeps.pip(n) Anport tine from threading import Thread, Timer, Lock rom queue inport Queve Amport randoneo ID; 6589695 resulta = queve() work.q = queve() work list = (1 1 The worker thread pulls an item from the queue and processes it def worker() while True: stem = worka.get() o_work(item) work_a.task done() #pause while until current work. task has completed ef do vork(iten) ‘et submit query process results ond add result to Queve ‘result_g.put( wait_randon(iten) ) ‘def watt_randon(t): ‘tine. sieep(t{1]) print( "Finished task ()' format(t{0})) def nelo(): rine("hello, world”) ‘loading up our work.q and work_List with the sone random ints between 1 ond 10 ‘ine total for i An range(se): xX = randos.randint(1,16) ‘ine total i= work 4. put((4,%)) work 1ist. append (44)) work.a.qsize() Aen(work 14st) ‘citine print("This should take {) seconds’ format(tine_total)) for w in work list: wat randon(w)oo ID: 6689695, oxtine for 4 in range(s): ‘t= Thread(target-worker) daemon = True # ehread dies when main thread exits. If we don't do this, then the threads wilt continue to 1 “Listen” to the sork.q and toke ‘tens out of the work.q and automatically process as you # stick more ‘tens into the work a testart() 4 you have to stort o thread before {t begins to execute work.q.jein() # block until alt tasks are done (U) You can also use the Timer class to specify that a thread should only kick off after a set amount of time. This could be critical if you need to give some other treads a head start of for various other reasons. Remember, when we are doing threading you have to keep timing in mind! soit ine 1 stupid LietLe exomple t= Tiner(5.0, hello) ‘L-daenon = True RLstart() # after § seconds, “hello, world" will be printed (U) You can mix these. The output below will most likely look like a bucket of crazy because threads execute (sort of) independently. ‘oading up our work q and work Uist with the sone random ints between 1 and 10 for 1 in range(se). x= random. randint(2,10) work. a. put ((4,%))Doe ID: 6589695 satine for § in range(s): ‘t= Thread(targetoworker, ) ‘daenon = True testart) = Tiner(5.0,helo) ‘eL.daenon = True a start() # tL ‘WiLL probobly* print “hello, sorta’ before alt the other threads Finish, # or dt might not {t depends on the work q contents work g.Join() block untiL alt tasks are done (U) Subprocesses (U) For most subprocess creation you will usally want to use the subprocess.run() convenience method. Please note, if you wish to access the STDOUT or STDERR output you must specify a value for the stdout and stderr arguments. Using the subprocess.PIPE constant puts the results from STDOUT and STDERR into the CompletedProcess object's attributes. Import subprocess conpleted = subprocess.run({"1s',‘-1"], stdout-subprocess. PIPE, universal_newlines=True) rint("ARGS:", completed. args) ‘print ("STOOUT:”, completed. stdout) Drint(*STDERR: “completed. stderr) Drint(*return code:" completed. returncode) completed = subprocess.run({Is", ‘nosuehfile" ],stdout- subprocess. PIPE, stdenr-subprocess PIPE, universal_nevlines-True)Bo ID: 6559698 Drint("return code: completed.returncode) snport * syntax. Other things __init_:py commonly includes are import. statements for subpackages, comments/documentation, and adeiional code that glues together the subpackages. {U) Since our module is small and we have nothing to initaize, we'l leave __inst_.py empty fwe wanted, we could actually put al the ‘awesome py code in init__.py itself, but that can be confusing for other developers. We could also define: aus (Awesome, “c001") (U) But since those would get imported anyway, we dont need todo that. You only need to define _a11__ fhe package is complex enough that {you want to import somethings and not others or ensure that subpackages get imported (U#FOUO}For another example, see the __\ot_9y fle forthe parserror module on GitLab,Boe ID: 6559595 2. (U) Make the package into a distribution (U) Atthis point, you could tar or zip up that package, give it to someone else, and they could extract it and use it. But what directory should they ‘ut the package in? What if your package depends on other modules to work? What ifyou change the package and want to keep track ofthe version? These considerations suggest that there is some package management that needs to go on. Python does have the pip package ‘manager to handle alot ofthat. So how do we distribute the package so you can just pip install avesone ? (U) First we need to add another layer of stuff around our package to help set it up for instalation, awesone/ README. awesome init__.py - Gresone.py setup.cfe ~~ setup.py (U) You can s2e that our original package directory has been moved down to a subdirectory. Above it we have a new project directory by the same name, and alongside it there are afew more files. Let's look at each (U) setup.py (U) setup py is the most important fe. It mainy contains a setup() function cal to configure the distribution. Its also used to run the various ‘packaging tasks via python setup.py . The setup() function comes from the setuptools package, which is not pat ofthe Python standard library. You may need to pip install setuptools fist to use it. (setuptools improves the legacy distutils package that is part of the standard liorary andi the oficialy recommended distnbuton too! these days.) (U#FOUO} Our setup.py looks lke this:oc ID: 6589695 from setuptools import setup setup erston="1.0 description-"(U) An avesone module for avesone things", Jong_description-open(*READHE. nd", 'r").read(), ord https: //eitlab.col.nsa.ce, gov /pyton/avesoneg18*, buthor="COnP3321°, Suthor_enail-comps32iens seripts-(1,] packagese["mecene’, 1] package dsta-(] Anstall_requires( i se.400 1 (U) Ths just scratches the surface ofthe arguments you can give to setup() . You can find more deals onthe outside from the Python Packaging ‘Authority (inttps://packaging.python.org/tutorials/istrituting-packages/ ). For example, you can specify which versions of Python are compatible using the classifiers and python requires options, specify dependencies with install_requires and other * requires options, ‘and include non-code data files. (UIFOUO) For another example, see parserror's soo (U) setup.cfg (U) setup.fg is an IN file that configures some defaults for the setup.py options. Ours is fairly simple: [metadata] description-File = README.na (U) Ifyou are using sheer , you may also want to set this if your code works for both Python 2 and 3: (U) README.mdoc ID: 68, {C}README mdi usta Markdown fle that ges an overview of what the package is for and describes how to install and use i (You may also see README rst les instead. The st stands for restructured text, which isa popular Python-specific way to format text.) 3. (U) Push the project to GitLab {U#FEYO) OF course, wit a projec ke this, you shouldbe using version control And since the nsa-pip server tes into GitLab, youll need to use ‘Gi. Learning Git could be a course in its, So weil just cover the basics here. (U) For those who are unfamiliar, Gitis a distributed version contol manager. Its version contol because it allows to to track and revert changes in ‘your text fies easiy in a Git repository. You can view the history of your changes asa tree, and even branch off from it and merge back in easly if You need to. Its disinbuted because everybody who has a copy ofthe git repository has a full copy ofthe history wih al the changes. And its ‘2 manager (ike ip is a manager) because it manages all the tracking itself and gives you a bunch of commands to let you add and revert your changes. a. (U) Install and configure Git : 7 another Linux sytem, you'can yon instal git (or CentOSIRHEL) or apt. probably wantto use Gt Bash or TeriseGt or something sma Stora Windows crectons hice (UsF9U} Ate Gis insta, you wi eed to confine it and SSH to conect ota. On LABBENCH, the east way od thas oun tis (Ruby notebook oseun Gi. and SSH in upyieocker, Ktakes caret ‘+ making a gitconfig, + making an overall tignore, and + making an encrypted RSA key pair based on your Pk cr for use with SSH {(U#F249} you ae on another system. you wil aed ake cae of hose thingsyoursel. For ips, see the oni salu nstucions ox ion Wit b. (U) Add version control to your project (U) Now to work! Enter your local project rectory (i.e. the top level where setup py lives) an run git Anit . That will tur your project into aa Git ‘epostory and get Git ready to track your files. (Note that it doesnt actually stat racking them yet-you have to explicily tl t what to track frst.) (U) Next, since this is a Python package, you probably want toa .gitignore fle to your project directory containing:Do ID: 6589695 buitd/ "oye (U) That wil to Git to ignore temporary fles made when you run setuptools commands. (U) Now our project structure should look something ke ths: swesone/ [n> weit [oo Cosgit str...) ~ pitignore ~ awesone/ _init_oy ‘recone py setup. cfg setup.py (U) Next, you can start tracking all these files for changes by running est aad * sit commit -m “Initial commit.” c. (U) Make a corresponding GitLab project (U) Congratulations! Your package is now under version control. However, you have the only copy ot, 60 your local copy goes away, your code 's gone. To preserve it and enable others to work on it, ou need to push it to GitLab. (UIFOUO) The frst stop is to make a now GitLab project 4. (U) Visit the nsw povject nage on GitLab 2 (U) Enter your package name asthe Project path (0.9, svesone ) 3. (UAFOO) Enter the overall classification level ofthe code and files in your project. 4 (U) Leave the Global acooss level sat to Reporter. Thal wil allow others to bath clone (copy) your code and fle issues if there are problems With your package. See the Gil.ab permissions chart fra full description ofthe access levels. 5. (UFO) Choose the Namespace. By default it will make a personal project under your name and SID. Ifyou belong toa GitLab group {and have the right permissions), you can also add the project to that group. Consider joining the Pinon group and adeing your project there. 6. (U) Add a short deseripton of your package, if you want 7. (UAFEWO) Select a visibility level Since NSA GilLab uses PKI, there is no diference between “intemaf" and “publi.” 8 (U) Hit"Create Project.”oe ID: 689698 {G)Aferwards, you can copy the URL from the main page of your new project and putitin the uel argumentto setup() In setup.py d. (U) Push your code out to GitLab (U) Your new project page should have some instructions on how to push code from an existing flderor Git repository. Near the top of the page you should soe a box with "SSH" highlighted tothe left anda it@gitlab.coi.nsa.e.gov... address inthe box tothe right. Copy that address. Then, 41. (U) ¢ to the top level of your project 2 (UMFOUO) it remote add origin gitGpstlab.col.nse.se.gov... (using the git address you just copied) 3.(U) eit push -v origin master (U) Afterwards, ifyou visit your GlLab project page, you should see a list of your projec les and the rendered contents of your README. 4. (UHFOUO) Create a tag for the package release. (U) Back in your local repository, tag your local project with pip- and the release version and push that tag to GitLab For example, version is 0.4.0, run ait tag -2 pip-0.1.0 -m “Releasing to the pip repo fait push origin pip-0.1.0 (U) ldealy version numbers should comply with Python's PEP 440 specication. n plain English, that means they shouldbe of the form: ajor. Minor micro | \- changes every bugeix 1 1 ' | \- changes every new feature ' v changes every tine backwards-conpatibility broken 5. (UHFOUS} Create distributions (U) Source distribution (sdist) and Whee! cstbutions are both collections of files and metadata that can be installed on a machine and then used a Python library. Wheels are considered a “bul distribution” while dist needs one extra build step, athough that i transparent toa userifyou ‘are using pip to install. For most use cases, you want to build both an edit and bast wheel, upload both, and lt pip work out which one o use (almost always the Whee!)oe 1D: 6689695 (G)An example of building and publishing the distributions is as folows: ppp install setuptools wheel twine pythons setup.py sdist bdist wheel a. (U) Upload package to NSA-Pypi with Twine-- (U) wine is @ Python library developed by the same team that maintain Pypi. It uses requests py to make secure (nips) uploads of Python ‘packages, and offers some commandline arguments to make it easy to specify what reposior, lent cet, ca bundle, and usemame/password to se. The usemame and password can be blank (or anything you want) for uploading to NSA-Pypi, because NSA-Pypi wil use your PKI and. ‘Caspot to determine authentcaton/authorzaton (U) Note thatthe NSA-Pypi server supports XPE with Labbench, so using twine there on Labbench does not require uploading your personal cat. Innor-Labbench environments, the folowing command should work twine upload dist/* + ‘Fllenane = “nb classification data.cey” AF not Filename in o$.14stdie(): File content = bucket .get(Filenane). content open(Filenane, "wb').write(File_content) features labels = pd.read_csv(filenane) set_index("té") ef preview(aF, nan 4 sampled! af = oF. sample(n-nrows) print “Preview of {):".fornat(nane)) ‘dssplay(@Fnead(nrows)) fata", rows-3, sampled-True): preview( features labels) print("statistical summary of data:") display( Features labels. descrsbe()) print("Datatypes in dataset:") 0.8", inplece=True) ‘in probability = max probs_éf[ probability” ].min() inax_probs_dfl'correct")} = max_probs_df{'actual'] == max_probs_df{ "prediction" } possible_prob scores = np. Linspace( startenin_probability, stop-.95, run-ine(np.cetl((.95 - min probability) * 208)) ) ‘rob_to_average_score « pd.DataFrane({‘prob_score*: possible_prob_scores}) average_scores ~ defaultatct( list) for score in possible_prob_scores: average score = np.nean(nax_probs dF query("probability >= (0)".format( round(score,3) )9E" correct) average scores[ ‘total! ].append(average_ score) ‘rob_to_average_score{ ‘average score") = average_scorestotal"] plt.Figure(dpi-100, Figsize-(20,5)) plt.plot (prob to_average_score{‘prob_score' J, prob_to_average_score| ‘average_score']) plt-xlabel("Probability cut-of (Lover bound)") plt-ylabel("Average accuracy percentage")oc ID: 6589695 Plt -title(“Average accuracy percentage of predictions above a certain probability score") plt.show() Distribution of Probability Scores ‘This ust shows the distribution ofthe probably scores. We want to see a skewed-ef distribution, meaning that most of the predictions are made with a high probability score. plt.figure(dpi=100, figsize=(10,5)) plt.hist nax_probs_df[ probability"), bins-20) plt.xlabel("Probability Scores") plt.ylabel (“count”) plt.title("Distributtion of Probability Scores from Predictions”) plt.show() Show Stats on Categories of Notebooks ‘This is just showing the statistics onthe categories of notebooks as of 14 November 2017-these are just based on the labels in our taining data, ‘not on the model predictions. fig, ax = plt.subplots(Figsize=(20,16)) bels[ “eategory" ]-value_counts() plot. ple( autopet="12.2P88", ‘ites "nbGallery Notebooks by Category as of 14 Novenber 2017", ax-ax) plt.show() Run Model on a New Notebook Jus enter the URL fora notebook in nbGallery, and see what category the model predicts for itee ID: 6559695 class Notebookextractor (object): def _tnkt_(seiF): Gelfs05 = requests_pki.session() self notebooks_url = ‘hetps://nbgallery.nsa.tc.gov/notebooks/* self. tooltip_finder = re.compile("(?:notebook has been|health score)") ef downioad_notebook(selF, notebook td): AF "/* in notebook 44: notebook sd = notebook £4.rstrip(*/") notebook sd - notebook sd. split("/")[-1] self notebook id = notebook 44. split(*-")[0] resp = self.ses.get( self.notebooks url + notebook id, headers-{ ‘User-Agent’: "Mozil18/5.0 (Windows NT 6.1; MOMGA; r:21.0) Gecko/28100101 Firefox/31.0", “Accept: "text/ntml, appLication/xhtml xml, application/xnl;9 » ‘ineout-30 > return resp def to_soup(self, resp): return Beautifulsoup(resp. content) dat extract tooltips(self, soup): feoltips dict for Link in soup. find ali(‘a', (‘class's ‘tooltips’, ‘title’: self. toottip_finder)): 4€* shared” in Uink{"tsCle"]= continve Af health score” An 2ink{"it2e"] health = re.search(=(fie)k", Link{ "tite" ]).eroups() fo) tooltips. sict{ health") = round(Flont¢health) / 109, 5) use ay » re.search("notebook has been (Qh) ~, Link “tstte"]).¢roups(9(01 ey = re-sub("(r2ed)$", =>, key) wey et ‘otal val = re.search(* Go fetin ‘tooltips dict{key] = int(total_val) unique. val = re. searen(“(\po)ousers", link tstle")) 5°, Link tit2e").groups(){2]oe ID; 6589695 AF untque_val: tooltips dtct{ unique ~ + key] = Ant(untque_val.groups(){0}) return tooltips dict ef extract toxt(self, soup): extracted text = [1 for a in soup. select(“snotebook0isplay")[0] FindAll 4 a.mane in [*hi*, "ha", "m3", "he, “hS*, “ps extracted text” append(a.text) return * ".Join(extracted_text) ef extract_description(self, soup) escription = soup. find nets’ ‘return description {nane': “description’))[ content] ef extract title(self, soup): title = soup.Find(“title’).text return title ef extract_classification(self, soup): ‘classification = soup.find(‘aiv', {‘class': re.compie("classtanner.¢*)}).text return classification det extract omer type( self, soup feroup = soup.Find('a", (re: re.comte(*H/erouns§.«°))) 46 grow: return "Group" return “User de struct lang(self, 509) Tong. tog ~ s00p.Find(ing', (tS lang» research Gb". Lana 9 return lang re.compile("This notebook 4s written in *)}) ‘eitle"]).groups()(0} lower) def extract(self, notebook 44, to_pandas-Trve): resp ~ self download natebook(notebook id) soup = self to_soup(resp) rb_dict = self_extract_tooltips(soup) rnb_dict[ notebook text] = self.extract_text(soup) nb dtet[ “description’] = seif.extract_description(soup) nb dict{"title') = self.extract_title(soup)oc ID: 6689695, nb_dict['classification'] = self.extract_classification(soup) nb_dict[ ‘omer_type"] = self.extract_ower_type( soup) rnb_dtct[ "Lang'] = self.extract_lang( soup) nb_dtet("4d") = selF.notebook id AF to_pandas ‘return pd. DataFrane( nb _dict]) set_index( id") return nb dict clase Notebookdodel (object): def _init_(se1f, 46 def snithalize rogex(self): Self.a = aF-copy() self class mappings = (TOP SECRET’: 3, “SECRET: 2, “CONFIDENTIAL self sealer = scaler self Features to_seale ~ features to_seale self. imputer = inputer self features to_inpute ~ features to_impute elf features to_encode ~ features to encode self encoded columns = encoded colums self validate colums() self initialize regex() self stenmer = Snowballstenmer( english’) self mapper = mapper self selected Features = selected features self all Features = List(training data) self. clf = best_clf + 'UNCLASSIFIED': 0) ‘self.pm_stripper = recompile” (?:Ak (K+) "> ‘self.o_word.chars = re.compile(*[*2-2]+5") self. strip_punct digits - re.compile(’({\ka-2]") Self. strip_extra_spaces = re.comptle("Ns{2,)") self. contains digits ~ re.conpile(’(.*ffs.*)") ‘self. strip_punct = re.compite(” (\bikI") validate_columns(self): for col in self. features to_scale + self. features to_impute + self. features_to_encode: Af col not in list(self. df): self.¢f[col] = np.nan def classification_to_level(self)PorID: SEBREPS axe aff‘classi fication level’] = self df{"classification'] apply (lanbéa x: x.split(?/)(0)) self. df[classification level! ] = self.df{ ‘classification level’ ]-apply(lambda x: self.class_mappings[]) Self. df.drop({classification’], axis-i, inplace-Trve) ef encode datetine(self, 2): Sstrp_string = "2¥-im-%d 2A: mm:%S AF Len(x) == 25: strp_string += * +0000" timestamp = datetine.strptine(x, strp_string) return (datetine(2017, 13, 14) - tinestanp).days ef datetine_to_days_ago(self, tinezone-True Af “created_at' in self.dF.colums and ‘updated at’ in self.df.colums: self-df{ ‘days_ago_crested’] ~ self.of[ ‘created at"].apply(lanbda x: self. encode_datetine(x)) self df ‘days_ago_updated'] - self df[ ‘updated at'].apply(lanbda x: self encode datetine(x)) self.df.drop(["created at", ‘updated st", axts-1, inplace-True) aise: self.dF[‘days_ago_created'] = © self.dF{ ‘days_ago_updated'] = © def calc ratio(self, field): Af fleld An self.aF- columns: self. df{ ‘unique {0} to_(0)'.format(Field)] = selF.dF{unique_' + field] / self.df{ Field] ease. self.df[field] = 0.0 self-df[ ‘unique’ + field) self-df{ ‘unique (0) to { format(Fleld)] = 0.9 def calc. ratios(seif) self cae ratio( runs’) self. cale_ratio( dowloads") self cale_ratio( views") Af self. aF['days_ago_created"].values[2] =: self.df{ ‘updated to_created"] = 0.0 else: self.dF{ ‘updated to_created"] - self. df 5900 updated’) / selF. af ef scate_features(self): self df[self. Features to_scale) = self. scaler.transforn(self.dF[ self. features. to_scale]) ef inpute_nissing( selDvcIP: 69095 are aftaelf. Features to impute] ~ self smputer.transfora(self.df{selFfeatures_to_sapute]) det encode _categories(self): self.d¥ ~ pd.get_dumies(self.d, column for col in self. encoded column Af col not in list(self.f self.af[col] = 0 If features_to_encode) ‘def strip_portion_sarkings(self): self. text_features = [col for col in List(self.dF) AF self.dF{col] type == np-object] for col sn self.text_features: self.df{[col] = self.df{col].2pply(Ianbéa x: self.pm_stripper.sub("*, x)) def normalize text(self, text): normal ~ text. lower) 1 remove any words contatning digits normal - “ *.Join({selF.contains digits. sub(" ", word) for word in normal. split(” *)}) 4# remove words that contain no word characters(a-2) normal - “ *.Join({selF.ne_word_chars.sub(* ", word) for word in normal.split(* ")]) 4 renove oll punctustion and digits normal = self strip_punet_aigits.sub(**, normal) # renove alL punctuation ‘mornal = self.strip_punct.sub(* *, normat) # replace consecutive spaces with a single space normal = self.strip_extra_spaces.sub(* °, normal) 1 renove Leading and tratLing whitespace normal = normal.strip() return noraal ef normalize text_features(self): for feature in self.text_ features: self.df[feature] = self. df{ feature] .apply(self.normalize text) ‘det stom text(self): for col in self.text features: self. dF{col] = self.df{col] apply(lanbda x: * *.join([self.stemmer.sten(word) for word in x.split(" °))) ‘def F411_ntssing_coluns(sel¥) for Feature in self.all features: AF feature not in list(selF.¢f): self.df{Feature] = &POIDS RPS sea gf « self. df Fi1I00(0) def transforn(self) self, classification to_level() self datetine todays g0() self cale_ratios() self scale_features() self impute sissing() self encode categories() self. strip_portion_markings() self normalize text features() self sten text) self. 111 missing colums() self. transformed = self.mapper.transfora( self dF) ef predict (self): return {‘predicted class's self.clf.predict(self.transformed)[0]} ‘det predict_proba( self): probs = self.clf.predict_proba(self. transformed) mmax_prob = round(np.rax(probs), 3) max_class = self.clf.classes_{np.argnax(probs)] return {"predicted class": max_class, ‘probability’ rmax_prob) ‘def transform predict(self, probability-False): self. teansform() AF probability: ‘return self.predict_proba() return self predict() ‘otebook.url = input("Enter the nbGallery URL for 2 notebook: ~) pe = Notebookéxtractor() ‘b_dF = ne.extract(notebook_url) trode] ~ NotebookModel(nb_¢F) prediction = nodel. transform predict(probability-True) Dprint("The predicted category for {) is {) with a probability of {).".format( notebook url, prediction( ‘predicted class'].tithe(), prediction{ ‘probability'}))Doc I 68969 COMP3321 Day02 Homework - GroceryDict.py Smt ea witty ree 300 10 — as (U) Homework for Day 2 of COMP3321 (U) COMP3321 Day02 Homework - GroceryDict.pyeID: 6689685, ‘# Grocery Uist yGroceryList = ["apples", “bananas”, “milk”, "eggs", “bread”, hamburgers", "hotdogs", "ketchup", “grapes”, tilapia", “sweet potatoes", "cereal", “paper plates", "napkins", “cookies”, “Hee creas”, “cherries”, “shagpoo"] vegetables - ("sweet potatoes", "carrots", "broccoli", "spinach", ‘ontons", "mushroons", "peppers"] fruit = ["bananas", "apples", "grapes", “plunbs", "cherries", “pineapple”) cold_itens = ["eges", “milk”, "orange juice", “cheese”, “ice crean"] proteins = [*turkey", "tilapia", "hanburgers", "hotdogs", "pork chops", "han", “neatballs") boxed_itens - ["pasta", "cereal", ‘eatneal”, "cookies", "ketchup", “bread"] paper_products = ["tollet paper", “paper plates", "napkins", “paper towels") tolletry_itens = ["toothbrush", “toothpaste”, “deodorant”, "shampoo", soap") GroceryStore = dict({*vegetables":vogetables, “fruit”:fruit, “cold_itens":cold_ttens, “proteins":proteins, "boxed itens":boxed_itens, “paper_products":paper_products, “tolletry_itens":toiletry_itess)) myewGroceryList = dict() (U) Fl in your code below. Sort the items in myGroceryList by type ino a dictonary: ayNexGrocerylist The keys ofthe dictonary should be: rvegetables", “fruit, “cold itens", “proteins”, “boxed items", "paper products", “toiletry itens"] (U) Only use the crocerystore dict , not the individual tem lists, todo the sorting, Note:The keys for mMeucroceryList are the same as the keys for GroceryStore.Do ID: 6589695 print (my vegetable List: ", myNewiroceryList.setdefault( vegetables", 1ist())) print (my fruit List: *, mewGroceryList.setdefault("fruit", List())) print ("my cold item List: ", myNewiroceryList.setdefault( ‘cold items", 1ist())) print("My protein List: ", mesGroceryList setdefault( ‘proteins’, 115t())) print ("my boxed item List: ", myNlewGroceryList setdefault( ‘boxed itens', list())) print ("My paper product List! ", myNesGroceryList.setdefault( ‘paper products”, 1ist())) print ("my tolletry item List: ", ayNesGroceryList.setdefault(‘totletry tess", list()))ec 1D: 6689696 Password Project Instructions for COMP3321 ted 6 months ago by| ia COME 3521 Sac (&) Password Project Instructions for COMP3321 See ae (U) Password Project Instructions for, COMP3321 (U) Tse ar the password projectnstucins fr COMP3321[_——ou neo sea fe contarng your uncon) the Inctudor nota retabook. Flee shout be named SID pestwoTT RRS TT (U) Password Checker Function (U) Demonstrates the abilty to loop over data, utlizing counters and checks to see if all equrements are met. (U) White a function called password checker that takes as input a string, which is your password, and retums a boolean True ifthe password ‘meets the following requirements and aise otherwise. 1. Password must be at least 14 characters in length 2. Password must contain atleast one character rom each ofthe four character sets defined below, and no other characters. ’3. Passwords cannot contain more than three consecutive characters from the same character set as defined below. (U) Character sets: ‘+ Uppercase characters (string.aseii_uppercase ) + Lowercase characters (string-aseii_lovercase ) + Numerical digits (string. digits ) + Special characters ( string.punctuation )Bo ID: 6539696 YYou'may want to write mutiple functions that your password checker function cals. “Due: End of Day 3 {ef password checker password) This 4s my awesone docstring for ny avesone pessnord checker that the author should adjust to say something sore meaningful. return True (U) Run the folowing bit of code to check your passsiora checker function. I your code is good, you should get four (4) Trve statements printed to ‘the screen, 2 This 1s a good password print (password checker("abcABC1231@#abcABC1231 64 ‘rve) 1 Ts ts Anvaltd becouse the runs of sane character set are t00 Long print (password_checker( "abcde fgABCOEFGI2345671688%°8") == False) # This 1s invalid becouse there are no characters from string. punctuation print (password_checker("abcABC123abcABC123") == False) # TMs ts alts becouse &€ ts t00 short print (password_checker("aa861166") =~ False) (U) Password Generator Function (U) Demonstrates the ably to randomly insert characters into @ string that meets specific password requirements (U) Write a function called password generator that takes as optional argument the password length and retums a valié password, as defined in Password Checker Function. Ino length is passed tothe function, it defaults to 14, The following cade autine does not account forthe optional ‘argument. You must make a change to that (U) Donot use the passiord_checker function in your password_genenator . You can use it ater you get something returned from your password. generator for your own testing, butt should not be part ofthe function itself, Due: End of Day 5Doc ID: 6559696 ‘def password, generator (Length): This is my awesome docstring for ay avesone password fgenerator that the author should adjust to say something sore meaningful. return True (U) Assuming you have a valid password_checker function, use the folowing code to check your password_generator Iino False sprint, you are ‘900d. Otherwise, something is up. my_password - password generator() Af len(nmy_password) '= 14 oF not password_checker(ay._password): print(False) ry_passvord = password generator(25) AF en(ny_password) |= 25 or not password checker(wy_password): print(False) (U) Ifyou realty want fo testi out, run the following. If False prints, something is wrong ‘from random import randint for 4 in range( 10000) Af not password_checker(passord_generator(randint(24,30))): print(False)Doo ID: 6589697 Final Project Schedule Generator —___ (U) Little notebook for randomly generating a final project presentation schedule. Students will present every 30 minutes slarting atthe start time specified, with an optional hour blocked off for lunch, Import Dependencies Amport pydeps Spydeps.pip(('query_input']) Amport random fron datetine import datetine, tinedelta inport ipywidgets as widgets import re (Python display import display, clear_output ‘uery_Anput Asport QueryInput f f Run Random Generator! This uses a fandoncenerator class that inherits trom the queryrnput lass from the query_snput package we imported to make creating the widget box and extracting the values outa litle easier. Approved for Reoc ID: 6589697 lass RandonGenerator (Queryinput): def _init_{(self, title-"Enter data for random project presentation tine generator"): iuper (RandonGenerator, selF)._init_(title) self.default_layout = ("1_width': "260px', 'r_width': ‘4B0px', ‘r_justify_content* "box justify content”: “eenter*} def generate tines self): start tines - [J ‘for 4 in range(s90, 1500, 50): start tine = ste(1) start_time = re, sub(" AF Len(start_tine) => 3: start_tine - "0" + start tine start_tines. append(start_tine) return start tines "30" start_time) def random schedule(self, students, start tine, lunch tine-tone): start_tine = datetine.strptine(start tine, "") A Lunch tine: ‘unch_tine = datetine.strptine(lunch tine, “=H0N") ‘random. shuffle(students) for student in students: print(f"{student} will present at {start tine.stetine("00")}*) Sf lunch tine and start tine =- lunch tine ~ tinedelta(ainute start_tine += tinedelta(ainutes-26) ne start_time += tinedelta(sinutes=20) det subait(self, 6): clear_output (wait-True) self. validate snput() self extract_input() students = [Student.strip() for student in self.extracted_input[ student nases"].split(*\n") 4 student.steip()] self.randon_schedule(students, self.extracted_input{ ‘start _tine’], self.extracted input get( “lunch ine")) ef create_input_form(self, start_tines): self butld box(deseription="cb/>Enter student nanes, one per line:", Inane="student names’, requiredsTrue, widget_typenwidgets. Textarea, r_height="400px", self default layout)oc ID: 6689697 self. butld_box(description-"cb/>Select the start tine: , options-start_times, nane-‘start tine’, required-True, widget_type-widgets.Select, **self.default_layout) self bulld box(description-"Select a lunch tine (optional):", options=[Wone] + start tines, name-'lunch time’, required-False, widget_type-uidgets.select, "self default_layout) self build box(description-"", nane-'submit", widget_type-xidgets. Button, button text~'submit’, bbutton_click-self.submit, required-False, box justify content="center', r_border='solid 3px’, ovkdth="200px", _hedght='Sepx") ef cun(serf): start tines = self, generate tines(), self create_input_forn(start_tines) self display(border="solid 25x") 1 = Randoncenerator() re-run)