A Python Primer Ela mets tyACKNOWLEDGEMENTS... eee eeeee Sf 0TC PCE Cees eee seebhiicevie i alban eres | INTRODUCTION... Objectives and Goals ssh duucsasenaeuseiaegnevauesesamidedvesiniveressasuucntig Fipsradpisiaeriutecpehsiine ee men lS SASSER GF URS SNS eueaemsaeiedeonen Developing Geoprocessing WorkflOws................. al Structure of the Book Be Data and Demos.. ans Obtaining the Supplemental information nnn Required Software ... sineinvuiaebaricatern esa a aac cae Older Versions of ArcGIS cal? ‘lie... oie Author's Note .. 23 SECTION I: THE FUNDAMENTALS ..0...cccccccccesccccesssen. cocenienimrtenceashs: RAD Chapter 1 Python and ArcGIS... Overview .. Python and ArcGIS enone wn dd SALOU Parent TS TAIT WERE AVON oii ing oS Soelace cote desc swaedena ta toe eval alle ouk abdeale aa aca Python Development Environmentts...........cs. = Se = a= x= = — kel nnklUcrlUlhe OU = ee a ee ee ee eeIntroduction a os eae years Esri has supported the use of the open source scripting oat oe aocneniee its geoprocessing tools and functions within ArcGIS independent, so it serves as a i ipti : = good single common scriptin language for different operating systems as well as for different versions of 5 es As aa development moves forward, organizations and individuals wi ie a i. nai geoprocesses using multiple scripting languages such as Arc Macro Language AML™) and Avenue™, which a F re now bot pcearinhereribseiet both outdated and python is rapidly becoming the scripting language that geographic information systems (GIS) professionals and newcomers to GIS want and need to learn. They see knowing how to program, especially using Python will be beneficial to their career and help existing organizations that have a long history of scripting development to transition to more current geoprocessing standards, especially if the organization uses Esri software, which many do. Objectives and Goals This book is written for those who want an introduction to Python using ArcGIS geoprocesses. / Python Primer for ArcGIS is not a detailed text on Python. Others have already accomplished this task. References can be found throughout and at the end of the book. A Python Primer for ArcGIS will help newcomers te SiS and programming as well as those who are strong users of ArcGIS geoprocesses but do not yet have a solid knowledge or expertise in writing scripts. For those who have some background in programming, many of the concepts such as variables, loops, conditional statements, etc will be familiar and helpful in developing Python scripts. For those who do not, this book will serve as a Starting point to begin developing code using some of the basic programming structures that are commonly used in many of the geoprocessing tasks found and used within ArcGlS. A Python Primer for ArcGIS focuses on d Python script ina logical manner to primarily hat can be run both inside and outside of ArcGls. A bjectives within this book: developing geoprocesses an develop standalone scripts t Python Primer for ArcGIS accomplishes the following 0 A Python Primer for ArcG|S®ode developers at different levels to design las design logical coding structures that handling, troubleshooting processes, 1. Provide a framework for c logical geoprocesses as we include proper constructs for error logic, and scripting problems 2. Introduce common Python constru with ArcGIS geoprocessing tools 3. Assist the code developer to obtain help with Python, ArcGIS geoprocessing functions, while building their own code writing skill 4. Introduce some of the new functionality of Python and ArcGIS, such as the mapping and spatial analysis modules 5. Show how to integrate custom built scripts with the ArcToolbox™ 6. Show how to auto-run custom scripts cts and how they are implemented The primary goal of A Python Primer for ArcGIS is to provide the user an organized path to gain a solid understanding of the common Python elements used in ArcGIS as well as to demonstrate how Python can be used for the most widely used geoprocessing tasks. These tasks will make up the majority of the book's content and is the primary reason why the author focuses on developing standalone scripts. With a grounding in both Python structure and syntax and common ArcGIS functions, the reader will be able to apply this knowledge, skill, and ability to more complex scripting and geoprocessing tasks (e.g. Python dictionaries, arrays, functions or ArcGIS extensions, ArcSDE®, and specialized geoprocessing methods) while having a structured environment to develop geoprocessing and scripting workflows. Upon reading and studying the concepts in A Python Primer for ArcGiS and performing the demonstrations and exercises and answering the chapter questions, the reader should be able to design, develop, create, troubleshoot and successfully run Python scripts with multiple steps and multiple ArcGIS geoprocessing functions and methods. A Python Primer for ArcGIS*® et a ie ain h Se 28 FE15 Prerequisite Knowledge and Skill - eR pe of A Python Primer for ArcGIS should have a fundamental “ ers : a o : concepts such as geographic features (points, lines, and po ygons , feature classes, GIS geospatial data formats, data and attribute tables relational databases, records, rows, fields, columns, etc. as well as a fundamental understanding of ArcGIS, how it is structured, and how to use ArcMap™, ArcCatalog™, and ArcToolbox™. The reader sa user diuddiae know how to use some of the geoprocessing tools (Clip, Buffer, Select Layer by Attribute, Select Layer by Location, etc.) within ArcToolbox. Familiarity with ModelBuilder™ is recommended, but not required to use this book. Completing some of the Esri courses or similar introductory college GIS courses that use ArcGIS should be able to provide the requisite knowledge to get started with A python Primer for ArcGIS. The reader does not need to know how to program or know Python or any other programming language. This text will provide an introduction to Python and general Python programming constructs that can be used with ArcG|S geoprocessing functions. Problem Solving Problem solving is an important skill to develop in an analytical field such as GIS. As a GIS professional and college instructor the author has developed a variety of problem solving skills that he uses every day in his work, uses and communicates with colleagues and clients, as well as teach students in the classroom. These skills may take a variety of forms: 1. Determine and develop which GIS tasks are needed to implement a series of geoprocessing analyses 2. Determine which systems integration and data management tasks are required to generate effective and efficient operational business workflows 3, Develop which programming tasks and the order of the geoprocessing tasks are required to automatically process GIS data using Python A Python Primer for ArcGiS*16 em solving skill is not easy and takes time, practice, and often olutions to GIS problems and scripts. The ding and studying documentation, trying analyzing data and reviewing and Developing probl takes hours of research to create $ author spends considerable time rea out specific geoprocessing functions, interpreting intermediate and final results, develop and test specific workflows, and building simple to complex geoprocesses. One can think of this as a modified “Scientific Method.” Readers are encouraged to consult ArcGIS help, on-line forums, study other developer's code, and build a repository of scripts and samples for future reference. From a code development point of view, many of the above techniques are used, since the proper result cannot be achieved without writing the proper code (instructions) for the “computer” to implement the script. In addition, the author often creates written documentation (outside of in-line code documentation) that explain processes, methods, data input/output, and solutions to intermediate problems in “plain English” that can often be referred to as well as develop for more comprehensive and formal documentation that the author provides for others to use. The author encourages the reader to do the same. For those who enroll in the GIS classes at American River College, the author provides the opportunity to learn and develop problem solving skills. For those of who refer to this book, consult the sources above to develop your own problem solving techniques or contact the author for more information. A Python Primer for ArcGIS® MIRA Ole igi eeae ee, - Ff | = Ff ss = 7 Developing Geoprocessing Workflows Before a GIS person (or team) undertakes a geoprocessing problem, often a result, goal, product, or service is needed, desired, required, etc. These can take the form of creating a new data set, summarizing data to help make a decision, generating a set of maps to show results of geospatial analyses, providing a web service, or developing a process to manage and update data for a specific purpose. All of these tasks require some set of steps to generate the result and often require some kind of interpretation, analysis, and evaluation of data, intermediate results, and final results. The author has found it beneficial to develop a geoprocessing workflow before a project starts to outline or map out the data requirements, processing steps, intermediate results, and final results. (Oftentimes, in practice, the workflow is never developed or only developed after realizing the hard work a person or team has expended). Developing an outline or workflow before a project commences the GIS technician, analyst, or programmer can operate with a structured framework (i.e. the overall objectives and goals of the project). Having this larger perspective on a project or task, solid solutions, geoprocessing tasks, products, and services can be developed and be developed for the right purposes. In addition, an outline or workflow diagram provides documentation to the process that can be referred to, since many projects can take a number of weeks or months and a single person will likely not remember all of the specific tasks, data, and products, plus the GIS person will likely be working on multiple projects at any given time. These outlines and workflows can also be used in documents provided to other staff members, clients, or for internal documentation. The workflow can take many forms such as an outline of steps or a workflow diagram indicating the relationships between one step and another or how one step may be related to many steps. For example, one source dataset may be used in multiple geoprocesses. This kind of workflow is often seen when designing a geoprocessing model in ModelBuilder. The workflow can be fairly simple involving a small number of geoprocessing tasks, or it can be complex involving many data sources, processing steps, feedback (looping) mechanisms, and many outputs (geospatial data, tables, maps, web services, etc). A Python Primer for ArcG!S®18 les of actual outlines and workflows developed projects. The figure below shows an actual python script created by the author that processes an attribute domain for a feature class. Notice the comments (marked with a # sign) that briefly describe the specific geoprocessing task. The commented steps (sometimes referred to as “pseudo-code”) provide the framework for the script. The comments were actually written first before any specific ArcGIS geoprocesses were Created so that the general set of steps could be outlined and the author could determine if other geoprocessing steps were required and could be researched for syntax, parameter, data, and data type requirements. The outline was modified as required during the development of the script. The following figures show exampl by the author for specific tasks and Be Peete te Re Le Eat Fpmat Gun Qotions Widows Help fui LS Pe Bu: beu_paite_valie, pa see YP Rieece ie ie see eeL (UP. © ar iper stows = gp. SearchCursor [parts_cable, irows = gp. InsertCursor jsorted_parta_table) Srow = srows.Gext |} ifow = irows.NewRoy |) irow,Code = grow. Code irow.Description = srow.Description irows. Insert Row|irow) srow = Sscows. next (} wid : + Signa_fic + fit 22 Log, + signs fc 1 bs x <= 5: gp. RemovebomainFromF ieldiaigna_ tc, a" n 41 A Python Primer for ArcG/S®19 As another example, the figure below s i developed for . geoprocessing one oo a signs for the City of Sacramento. This workflow is used by GIS awe _ department of transportation. A GIS data management document ai am . the workflow that provide specific GIS data processing tasks that the oe ma: uses to update and manage the city’s traffic sign inventory. The GIS st ft : th city uses this workflow to discuss similar activities for otter city iancianel _ full discussion of the workflow and geoprocess implementatio be f r the Winter 2009 issue of ArcUser™ oak aite Oe Fe SE PE a oP ated that s outlined and documentation is cre in both cases, a general process i can be used for a wide variety of purposes. A Python Primer for ArcGIS®20 Structure of the Book with a description of Python and ArcG/S f the book discuss how Python is organized and structured and some of the fundamental requirements in order to develop code that is useful for ArcGIS. ModelBuilder is me ntioned and how it can be used to develop geoprocessing logic and test ideas before exporting the work to Python for further development. In addition, the subject of “getting help” is discussed, since additional assistance with ArcGIS routines, syntax, properties, and methods will likely be needed during code development for both Python structures and ArcGIS geoprocesses. The structure of this book begins fundamentals. The first chapters © he common Python elements and the ArcGiS Tools in Section |, Section II of the book focuses on how to develop Python code for many of the commonly used GIS tasks. These include such topics as querying and selecting data, creating data, relating data, working on subsets of data, and creating new output. This section also includes the implementation of the mapping module with ArcGIS for automating map creation tasks. Section Ill focuses on integrating and automating Python scripts within the ArcGIS Toolbox or as back-end scheduled processes. After a brief overview of some oft Most chapters will have a demonstration program that the reader can work on and develop using step by step examples. In addition, the reader can work on the chapter exercises to obtain more experience with developing code and to test the knowledge gained in the chapter and demonstration program. Most chapters have questions that focus on the important concepts. The author uses the following typeface conventions throughout the book: Street_CL — bold type typically indicates a feature class or table explicitly used in the text, demo, or exercise. Bold is also used to highlight ArcGIS Help documentation topics so the reader can easily find additional information provided by Esri. StreetName — italics type typically indicates an attribute field. It will also be used to indicate a published work. A Python Primer for ArcGIS®21 arcpy.SearchCursor () — Courier indicates example Python syntax within the text, demos, and exercises. - will indicate a required parameter for an ArcGIS tool or routine {optional_parameter} — will indicate an optional parameter for an ArcGIS tool or routine Data and Demos All of the data and demo scripts can be found at the author's website at »ww.jenningsplanet.com\PythonPrimer. The supplemental material will be organized in the following manner: \PythonPrimer\ChapterXX. Within each chapter the Data folder contains the data files required for the demo and/or exercise. Data files can be shapefiles, file, or personal geodatabase feature class or tables, or standalone tables (e.g. dBase format). ArcMap documents (.MXD) can be used as referenced or renamed for readers to modify and save their own work. A MyData folder is also provided so that readers can save their own work for demos and exercises. All of the data and ArcMap documents will be in ArcGIS 10 format. NOTE: The ArcMap documents reference the \PythonPrimer\ChapterXX structure above. If the reader changes this folder structure, the ArcMap documents provided by the author may need to have the source files in the Table of Contents revised to the new location. The scripts have been tested on Windows XP and Windows 7 32-bit operating systems. The reader may need to make some additional adjustments to data paths on 64-bit Windows systems. The data sources exist on the one of the following web sites or organizations: City of Sacramento — city related vector data and historical 1991 aerial photos County of Sacramento — parcel and street subsets CalAtlas — Landsat Thematic Mapper (TM) satellite imagery subset Refer to the text file associated with the supplemental data for more information as well as the websites at the end of the book. A Python Primer for ArcGlS®22 Obtaining the Supplemental Information tal information for this book at the author's The reader can obtain the supplemen — website using the following credentials (and are case sensitive): Www. /enningspr Username: PythonPrimer Password: PP4AGIS! Additional information will be provided on this website with any updates, changes, etc. Required Software w®, ArcEditor™, or Arcinfo® license, d version of Python for ArcGIS 10. GIS Programming course (Geog The user must have access to an ArcVie version 10 and Python 2.6.5, the supporte Students enrolled in the online introduction to 375) at American River College (http://wserver.arc..osrio edu/~earthscience/) can obtain a one-year student license of Arcinfo 10. Contact the author to check enrollment and validate academic status. Alternatively, the reader can obtain a copy of ArcGIS for Home Use at http://www.Esri.com/arcgis-for «hen! or from one of the ArcGIS books from Esri that comes with a CD and DVD. The CD contains the data, demos, exercises, and solutions; the DVD contains a 180 day fully functional copy of ArcView 10. Esri can be contacted to receive an evaluation copy of ArcGIS that can be used with this book. Readers with access to ArcGIS only need to copy the data referenced in the book to get started with A Python Primer for ArcGIS. Readers are encouraged to review their own data or acom pany’s data collection and practice writing additional scripts beyond the exercises and demonstrations provided in this text. nome indae A Python Primer for ArcGiS*% pee me SF 23 Older Versions of ArcGIS and Python Older versions of ArcGIS and Python can be used with this book, but the syntax and structure of some programming elements may differ considerably. The ArcMap documents (MXDs), files, and geodatabases will need to be opened with ArcGIS 10 and exported or saved to an older ArcGIS version. Itis recommended that the latest version of ArcGIS be used with the content in this book. See the Author's Note below and Chapter 1 for more details on ArcGIS and Python compatibility. Author’s Note As of the writing of this writing of this book, the current version of ArcGIS is ArcGIS 10 which uses Python 2.6. A number of readers will still be using ArcG\S 9.3.x and possibly 9.2 in their organization. This book focuses on the use of the arcpy™ module using ArcGIS 10 and Python 2.6. For code developers that require a Python script for use in ArcGIS 9.3.x or ArcGIS 9.2, Python 2.5 will be used with ArcGIS 9.3.x and Python 2.4 for ArcGIS 9.2, both of which use the arcgisscripting Python module and does not contain the mapping module. Script syntax will need to be written to support both the ArcGlS and Python versions. Code developers can consult the respective ArcGIS Resource Center web help om/content/web-hased-help and can also go to www} i 4 for some common sample scripts using previous versions of the arcgisscripting Python module. A Python Primer for ArcG!S®25 Section I: The Fundamentals Section | introduces the Python scripting language and how it relates to ArcGIS. Chapter 1 focuses on how Python can be used with ArcGIS and its relationship with ModelBuilder, since some readers may already have experience with Model!Builder. Chapter 1 also introduces the Python script Interactive Development Environment (IDE), called IDLE so that the reader has a basic understanding of where to write actual Python script. The reader is provided a high level overview of how to obtain help with both Python and ArcGIS as well identifying errors that are likely to occur when developing Python code. Chapter 2 focuses on ModelBuilder and provides a general overview of how ModelBuilder operates and how it can be used to develop geoprocessing logic and can ultimately be exported to Python script where it can be more fully developed. Chapter 3 reviews the primary Python constructs that will be used throughout A Python Primer for ArcGIS. This chapter introduces these concepts at a broad level where many of them will be more fully discussed in the context of their use with ArcGIS geoprocessing tasks and functions. At the end of Chapters 3 and 4, the reader has a chance to write a simple Python script as well as a simple geoprocessing script that will serve as the launch point to develop more complex geoprocessing routines using Python and ArcGIS. ‘A Puthon Primer for ArcGIS®ie! foals nae eee a lane en alana er eet: ES vee + tar eee bas nN ip bgp piainh- Fr27 Chapter 1 Python and ArcGIS Overview One of ArcGIS's primary functions is to perform geoprocessing and data management operations. These can include overlays (intersect, union, identity, spatial join), querying data (select records by attribute queries or select by spatial coincidence), and dozens of other geoprocessing functions such as those found in the ArcGIS Toolbox. ArcGIS can also be used to create and manage data in a variety of formats. These can include, create feature classes and tables, calculate attribute fields, update feature classes and attribute values, work with imagery and image processing tasks, and join data tables among others. Python, an open source, cross operating system platform scripting language, provides a means to perform many of the operations mentioned above as automated tasks and batch processes. Instead of a user having to manually select multiple tools, fill in parameters, and clicking the OK button, analysts can write Python scripts to automate workflow tasks that can be run by an analyst either through the Python script interface or through a custom tool in the ArcToolbox. Python scripts can also be executed to run at a scheduled time, so that the automated tasks can run after business hours or during low network impact. The ArcGIS help makes specific reference to two different uses of Python scripts: 1. Python Script Tool — this is a script that is written with the intent to be used in a custom ArcToolbox and used within an open instance of ArcMap or ArcCatalog. 2. Standalone Python script - this is a script that is written with the intent to be run or executed outside of an open instance of ArcMap or ArcCatalog and may be used ina Windows scheduler program to have the script run automatically without user involvement. This reference can be found in the ArcGIS Help under Geoprocessing— Geoprocessing with Python—Essential Python vocabulary. A Python Primer for ArcG!S*28 of this distinction because it will impact how some Python scripts are written and error handling is developed. A Python Primer for ArcGIS will focus on developing standalone Python scripts, but will spend time throughout the book commenting on certain scripting methods that can be used for scripts developed for the ArcGIS Toolbox. Chapter 10 is devoted to the development of a custom ArcToolbox, developing Python script to accept user t to the custom tool, and creating help documents input, binding the Python scrip is : for the custom tool. One of the major reasons for writing this book is to provide the reader an introduction to Python, use geoprocessing routines with Python, and be able to develop logical code and practice writing code to solve etitive tasks. Creating standalone geoprocessing tasks and automate rep programs provides the opportunity for the developer to obtain the full experience of developing a geoprocessing task strategy, logically designing and writing the correct syntax to perform the geop rocessing routines and tasks, and to problem solve syntax and scripting logic problems that ultimatly arise in almost any geoprocessing workflow or script. The author makes reference For example, a Python script can perform the following operations to create a custom data set. Create a new data set Add and define a specific number of attribute fields Query and select records from multiple input data sources Add new (features) records and calculate attribute fields to the new data set 5. Copy the features in the new data set toa different computer or server PUN In addition to the above, the Python script can be scheduled to run the above processes at a certain time and at a certain frequency (daily, weekly, monthly, etc) essentially automating geoprocessing tasks without user involvement.Python and ArcGIS Versions Certain versions of Python work with certain versions of ArcGIS and may affect how Python code is structured. Some of the methods are processed differently depending on the version of Python and ArcGIS. As mentioned above, ArcG!S 10, Python 2.6.5 and the arcpy module will be used in this book. Scripts developed for older versions of ArcGIS can be run within ArcGIS 10 and Python 2.6.5, provided the scripts are properly written and reference the correct version of the geoprocessing object, arcgisscripting.create() as opposed to the arcpy module used in ArcGIS 10. Some Python geoprocessing functionality has changed slightly with newer versions of ArcGIS and Python. See the following table which summarizes the geoprocessing object, ArcGIS version, and corresponding Python version. For more information consult the ArcGIS Help documentation for the respective ArcGIS version and review the Python related topics. A Python Primer for ArcGIS* ————————__—————— ee i~Geoprocessing Object Python syntax must use the required arepy syntax, structures, and methods import arcpy module No specific geoprocessing object required Some Python syntax must conform to ArcGIS 9,3 Python methods (such as lists or Booleans) 9,3" 2.5 import arcgisscripting module gp=aregisscripting.create(9- 3) ArcGIS 9.2 must use Python 2.4; ArcGIS 9.3 must use Python 2.5 import arcgisscripting module gp=arcgisscripting.create () It is recommended that newer versions (> 2.6.5) of Python are not used, since they are not supported by Esri and will likely not work. NOTE: If the Python programmer is developing code for one version of ArcGIS, but the end user will likely use a different version of ArcGIS, the script should be tested for that particular configuration and may require a different version of Python to be installed on that system. For example, if ArcGIS 9.2 is running on a server to run nightly script processes, Python 2.4 must be installed on that system and the Python script must be able to run successfully with Python 2.4 and ArcGIS 9.2 tool syntax. For developing code, any version of Python can be used, however the Python version must match the ArcGIS software for testing and implementation and use the proper import module. Different versions of the scripts may need to be developed depending on the version of ArcGIS and Python. *Indicates that if arcgisscripting.create (9. 3) exists, then certain Python structures (such as lists and looping structures) will need ta be written in a certain form and if the script is deployed on a different system, the machine must have ArcGIS 9.3 and Python 2.5 installed. If the arcgisscripting. create () is left blank, then the Python script can be written and processed with either ArcGIS 9.2 or ArcGIS 9.3, provided that the Python structures are written with the appropriate syntax for the specific version of Python. A Python Primer for ArcGIS®P 2? FPS S/ a How Python is used with ArcGIS The author is often asked if Python be used to change the look and feel of ArcGIS or create a custom toolbar or “button” to perform a special function within ArcGIS. The answer is “no.” However, a custom “tool” can be created and stored within the ArcToolbox which runs just like any other tool in ArcToolbox that is designed using Python script. Tools that use custom Python scripts often have parameters that are filled in like other tools. In addition, the new arcpy mapping module can provide some ability to create and generate cartographic output that previously required the use of Visual Basic for Applications (VBA) or VB .NET and a custom button and/or graphical user interface. Python is intended to help automate geoprocessing tasks often in batch mode or through a scheduled process. Historically, Arc Macro Language or Visual Basic for Applications has served this need. Currently, Visual .NET or cross-platform C++ is used to create custom ArcGIS applications or toolsets that require a user to interact with the ArcGIS interface or mapping environment. Other resources are available to address these topics and are beyond the scope of this book. See the following website for more information Python Development Environments A number of scripting environments exists to develop Python code. One can develop code simply by using a simple word processor or blank text file. Notepad, WordPad, Word, or other word processor application can be used. Integrated Development Environment (IDLE)* When a developer installs the Python application from the python.org site, a Python editor space, interactive development environment (\DE) - (IDLE), is available that allows for color coding of key words and some simple tools to assist the developer create and write code. Often when an existing script is opened with Python IDLE two windows appear: A Python Primer for ArcGls®