Planning and design for Python

Introduction
Code planning and design is an important first step in a coding project. Before starting to code, it is important to think about the requirements for the code
and design how it is going to be implemented.

One implementation decision is which programming language to use. Is Python the right tool for the job? In addition to the Yammer group, there is a wealth
of information online to help take you through getting started with Python, including installation requirements.

Getting started with Python

Depending on your previous knowledge there are many resources out there to help beginner coders and experienced coders understand if Python is the
appropriate technology for your project.

The W3resource Python tutorial front page provides an overview of why you might like to use Python and provides useful pages on Python features
including Python 2.x vs. Python 3.x, syntax, variables, if statements, looping, data types (including lists, dictionaries, tuples) and Python modules.

• The Essential Python Cheat Sheet provides a great (and brief) overview of Python including the differences between Python 2.x vs. Python 3.x,
looping in Python, Python data typing and operators, Python functions and how to use Python libraries.
• Python for data science cheat sheet provides a crib sheet of key bits of Python syntax and Python data types and methods that are likely to be
useful for data processing.
• Dive into Python 3 is a free book available online. A Python 2 version is also available.

The Python wiki provides useful links to Python tutorials for non-programmers, and for programmers including links to useful resources about moving to
Python from other languages.

All Python development should be stored and managed within the Mott MacDonald Global organisation on GitHub. The Mott MacDonald Python repository
template shall be used for every Python repository on GitHub. This GitHub template contains important documentation and structure to get started with
Python.
Installing Python
• The latest version of Python should be installed from Service Now. Microsoft Visual Studio Code should also be downloaded and used as the code
editor for your Python scripts due to its embedded Git control. Microsoft Visual Studio Code is freely available on Service Now. You may also use
other versions of Microsoft Visual Studio, such as Pro or Enterprise, depending on your specific needs.
• If you are wanting to use Python for data science, analytics and machine learning, Anaconda is a good option. Note that as of 25th February 2022,
Anaconda is no longer free to use for companies with over 200 employees. You may request a paid license for Anaconda via Service Now.
Alternatively, Microsoft Visual Studio Code does offer a good solution for data science projects. Refer to this tutorial for details.
• If you are going to use Python with ArcGIS Desktop (ArcMap), it requires Python 2.7.18 and Numerical Python 1.9.3 which comes
installed with ArcMap. ESRI have no plans to switch support to Python 3 for ArcMap.
• If you are going to use Python with ArcGIS Pro, conda and the ArcGIS API for Python should be installed on your system (these come installed with
ArcGIS Pro). Setting up vs code to work with the ArcGIS Pro conda requires some environment adjustments on your machine see::
o ArcGIS Pro Python Interpreter Setup with Visual Studio Code
o GIS Python Tutorial: ArcPy Coding with VS Code and Conda
o 4 Arcpy ArcGIS and VSCODE
• You can have more than one version of Python installed on your computer so there is no reason not to have both installed.
• The ESRI and Anaconda Python distributions come with many useful Python libraries already installed. However, if there are additional libraries you
need, you may install them separately using ‘pip’, the official package installer for Python, or ‘conda’ if using Anaconda.

Requirements capture
Requirements capture is the process of understanding, documenting and testing the desired functionality and behaviours of a system (or piece of code).

They should be an unambiguous and measurable statement of what is required. How the requirements will be achieved (i.e. implementation decisions such as
which programming language will be used) are not part of requirements capture.

The degree of requirements capture necessary will depend on the scale of a project. However, even for a small piece of code it can be useful to think about
what the requirements are.

Designing your code

Software design takes as its input the requirements identified during requirements capture. The output should allow the software to be implemented in code
with minimal further design.
Software design focuses on the system at a number of different abstraction levels. At the highest level it should seek to conceptualise the system to be
implemented.

The aim then is to break the conceptual model of the system down into a series of smaller components (modules), their interfaces and their interaction with
each other.

Software design should understand how the system components address the requirements. The logical structure and implementation of each component (i.e.
how they will be coded) can then be considered separately.

Using Python with other software packages

Python interfaces well with commonly used software packages including ArcGIS and Excel.

Python and ArcGIS

Useful online resources for getting started using Python with ArcGIS include ESRI Python for Everyone free online tutorial and this online course on using
Python with ArcGIS. The GIS community Yammer group may also provide help and advice.

Python and Excel

Xlwings is a useful Python library which allows you to use Python as a backend to Excel (an alternative to VBA). The advantages of using Python over VBA
include:

• Xlwings offers much the same functionality as VBA including allowing you to call Python code from Excel using buttons.
• Make Python user defined functions (UDF).
• Reading data from one (or multiple) worksheets into Python for data processing (including using named ranges), and to write data back to an Excel
workbook.

Refer to Xlwings docs for options on how to install this package.

Developing for Python
Best practice guidelines
Best practice coding guidelines and IMS requirements should be followed when undertaking any coding project using Python. These guidelines promote the
creation of good quality code, provide consistency across practices, and make it easier to check and reuse your code.

Ideally, the Python style guide should be followed when creating Python code.

These guidelines promote consistency and readability are more important. Use best judgement and do not break backwards compatibility just to comply with
these recommendations.

There are various automated tools available which check (or even actively tidy) your Python code for compatibility with the latest Python style guide. Two
such tools are:

• pep8
• PythonTidy - this actively tidies your scripts

IMS requirements
The IMS requirements for code depend on the scale and intended purpose of the code. These are summarised in the figure below.
Interactive coding environments
Jupyter Notebook is an excellent interactive coding environment that makes it easy to share code and can aid code development, testing and
debugging. Jupyter notebook may be installed using Anaconda. You may request a paid license for Anaconda via Service Now. Alternatively, Jupyter
Notebook may also be installed with pip, Python’s package manager, instead of Anaconda. You may use an IDE such as Microsoft Visual Studio Code, which is
freely available via Service Now.
Useful Python libraries
Python has many useful libraries that provide functions and classes that can make your life easier when starting to code.

o Numpy
 Multi-dimensional array manipulation package used fairly extensively in geo-spatial work.
o Pandas
 Excellent library for data analysis and manipulation of numerical tables and timeseries (a good cheat sheet here)
 Provides good date/time manipulation and easy reading/writing to csv files and Excel.
o Matplotlib
 Python data visualisation library (a good overview with some examples here).

Debugging
No matter how well you've planned and implemented your code it is likely that you will need to do some debugging during the development.

It is often easiest to interpret Python error messages from the bottom up. The last line indicates the type of error that occurred. The lines above provide a
traceback indicating where (in which file or function) the error occurred. If the function in which the error occurred is called by another function, then the
details of this calling function (and the line in this function which called the erroring function) are also displayed. Further tips on debugging Python can be
found here.

Get help
If you run into a problem you can't solve, there are lots of places you can go to get help.

• Yammer groups that may be able to help include:

o Python
o GIS Practice community
• Stackoverflow and the Python forum are useful online help forums.
• As a last (or maybe even first!) resort, Googling "How do I do x in Python" will normally get you to the right answer eventually.
Please be aware that most code you find online will have some sort of licence associated with its reuse (including that taken from help forums
e.g. StackOverflow).

Online forums should be used as a learning aid rather than a place to copy code from directly.

Testing and deployment for Python

Testing, checking, and authorising
The scale and purpose of your code will determine the minimum testing, checking, and authorising requirements for your code before deployment.

These requirements should be considered in parallel with any project-specific requirements.

The level of testing, checking, and authorising required for your code depend on the scale and intended purpose of the code as summarised in the figure
above.

The Guidelines for use of Python for decentralised development provide further details on testing, checking and authorising.

Documentation
For anything more than the simplest script it may be important to provide some form of documentation.

The technical details of the code (such as the purpose of functions) should, as much as possible, be incorporated into the source code.

However, it may be appropriate to provide documentation for end users detailing how to use the code.

This should be as concise as possible - if pages of documentation are required to explain how to do something it may be more appropriate to redesign the
code to be more user friendly.

Bug and code improvement logging

If your code is likely to be shared with other users, it may also be necessary to keep a log of bugs and suggestions for code improvements.

We generally recommend as a first choice using a bug logging system such as JIRA if it is available to you for logging defects
If this isn't available - this could also be done using a spreadsheet or Word document which should be stored in a backed-up location.

The Guidelines for use of Python for decentralised development provide further details.

Finally, if you've followed all these guidelines and recommendations, you've probably created an awesome piece of code with accompanying documentation
that is stored on the Mott Macdonald Global GitHub organisation. If it doesn’t contain confidential project information, consider sharing it with the wider
GitHub community by creating an open-source repository for your code.

Remember the software lifecycle doesn't end at testing and deployment, it is an iterative process.

The next step is to go back to planning and design of the next, new, and improved version of your code.