EODP Environment R1.0

Environment
Earth Observation Data Processing
Date: 14/09/2020
Revision: 1.0
Name Function Signature
Prepared by
Approved by Lucia Soto Coordinator of EODP lsoto@ing.uc3m.es
Authorised by
MASTER IN SPACE ENGINEERING

1 of 115
Date: 14/09/2020
Environment Revision: 1.0
Subject EODP
Revision Record
Modification Change Description Date
Review
reference
1.0 - First issue 14/09/2020

2 of 115
Date: 14/09/2020
Subject EODP
Table of Contents
1. Introduction _____________________________________________________________ 10
2. References ______________________________________________________________ 11
2.1. Applicable Documents __________________________________________________ 11
2.2. Reference Documents __________________________________________________ 11
2.3. Acronyms ____________________________________________________________ 11
3. Setting up the Work Environment. ___________________________________________ 12
3.1. Install Oracle VM Virtualbox _____________________________________________ 12
3.2. Import the Virtual Machine ______________________________________________ 17
3.3. Settings _____________________________________________________________ 23
3.4. Run the Virtual Machine ________________________________________________ 32
3.5. Guest Additions. ______________________________________________________ 34
3.6. Shared folder. Accessing the files in the VM from the host OS. ___________________ 38
3.6.1. Create a folder in the host OS ________________________________________________ 38
3.6.2. Update the shared folder settings in the VM ______________________________________ 39
3.6.3. Mounting the shared folder ___________________________________________________ 42
3.6.4. Mounting the shared folder - script _____________________________________________ 43
3.7. Close the VM _________________________________________________________ 43
3.8. Ubuntu updates _______________________________________________________ 44
3.9. Test COTS installed in the VM ____________________________________________ 45
3.9.1. Dolphin (file explorer) _______________________________________________________ 46
3.9.2. Notepadqq (text editor) _____________________________________________________ 47
3.9.3. Panoply (NetCDF viewer) ____________________________________________________ 49
3.10. Tips for working with a VM. _____________________________________________ 56
3.11. Shrinking the size of the VM ____________________________________________ 57
4. Command Prompt or Terminal. ______________________________________________ 60
4.1. Open a window and new tabs ____________________________________________ 60
4.2. Some basic operations __________________________________________________ 61
4.2.1. Change directory and see contents _____________________________________________ 61
4.2.1.1. ‘cd’ to a directory, ‘ll’ to list contents ________________________________________ 61
4.2.1.2. The ‘.’ and ‘..’ directories _________________________________________________ 61
4.2.1.3. ‘pwd’ see which directory you are in ________________________________________ 62
4.2.2. Making it easier to input a command ___________________________________________ 63
4.2.2.1. Autocomplete with Tab ___________________________________________________ 63
4.2.2.2. Arrows Up and Down for commands ________________________________________ 63
4.2.2.3. Reverse search Ctrl+R ___________________________________________________ 63
4.2.3. Files and Directories ________________________________________________________ 64

3 of 115
Date: 14/09/2020
Subject EODP
4.2.3.1. Create a file (nano) _____________________________________________________ 64

4.2.3.2. Create a directory (mkdir) ________________________________________________ 64
4.2.3.3. Remove a file (rm) ______________________________________________________ 65
4.2.3.4. Remove a directory (rm -r) _______________________________________________ 65
4.2.3.5. Copy a file or a directory (cp, cp -r) _________________________________________ 65
4.2.3.6. Move a file or a directory (mv) _____________________________________________ 65
4.2.4. Change permissions (chmod) _________________________________________________ 66
4.2.5. See how much a directory or file weighs (du -hs) _________________________________ 67
4.2.6. Compress or uncompress with tar _____________________________________________ 67
4.2.7. Symbolic links _____________________________________________________________ 68
4.2.8. Run a script or an executable _________________________________________________ 68
4.2.9. Asterisk (*) _______________________________________________________________ 69
4.2.10. Searching _______________________________________________________________ 69
4.2.10.1. Find in file (grep) ______________________________________________________ 69
4.2.10.2. Find a file or folder (locate, find) __________________________________________ 70
5. Implementation and Execution. ______________________________________________ 71
5.1. Python ______________________________________________________________ 71
5.1.1. The EODP source code in Python ______________________________________________ 71
5.1.1.1. Configuration __________________________________________________________ 73
5.1.1.2. Call __________________________________________________________________ 74
5.1.2. Python classes and “self” ____________________________________________________ 76
5.1.3. Saving the log in Python _____________________________________________________ 76
5.2. C++ ________________________________________________________________ 77
5.2.1. The EODP Geometry Module in C++ ____________________________________________ 77
5.2.2. Saving the log in C++ ______________________________________________________ 79
6. Version control with Git ____________________________________________________ 80
6.1. Create Account in GitHub________________________________________________ 80
6.2. Create a Project in GitHub _______________________________________________ 80
6.3. Clone your Project _____________________________________________________ 84
6.4. Git status ____________________________________________________________ 85
6.5. Git change branch to develop ____________________________________________ 86
6.6. Git add and commit ____________________________________________________ 86
6.6.1. Modify the README.md _____________________________________________________ 87
6.6.2. Summary and tips__________________________________________________________ 90
6.6.3. Add the sources ___________________________________________________________ 90
6.6.4. Commit the sources _______________________________________________________ 101
6.7. Git workflow ________________________________________________________ 104
6.7.1. Git for a single developer ___________________________________________________ 104
6.7.2. Git for teams _____________________________________________________________ 108

4 of 115
Date: 14/09/2020
Subject EODP
6.8. Git basic commands ___________________________________________________ 112

6.9. .gitignore file ________________________________________________________ 112
6.10. Exporting the commit history __________________________________________ 113
6.11. Exporting the source code _____________________________________________ 113
6.12. Saving the Git credentials._____________________________________________ 114
List of Tables
Table 2-1: Applicable Documents .................................................................................................. 11
Table 2-2: Reference Documents .................................................................................................. 11
Table 2-3: Acronyms ................................................................................................................... 11
Table 6-1: Some useful Git commands ........................................................................................ 112
List of Figures
Figure 3-1: Download page for VM Virtualbox ................................................................................. 12
Figure 3-2: Save executable for VM Virtualbox ............................................................................... 13
Figure 3-3: Installation of VM Virtualbox ........................................................................................ 13
Figure 3-10: VM Virtualbox .......................................................................................................... 17
Figure 3-11: VM Virtualbox. Import Appliance. ............................................................................... 18
Figure 3-15: VM Virtualbox. Destination folder for the VM. ............................................................... 22
Figure 3-16: VM Virtualbox. Import............................................................................................... 22
Figure 3-17: VM Virtualbox. Import............................................................................................... 23
Figure 3-18: VM settings. ............................................................................................................ 24
Figure 3-19: VM settings. General................................................................................................. 25
Figure 3-20: VM settings. System. ................................................................................................ 26
Figure 3-21: VM settings. System. ................................................................................................ 27

5 of 115
Date: 14/09/2020
Subject EODP
Figure 3-22: VM settings. Shared Folders. ..................................................................................... 28

Figure 3-23: VM settings. Shared Folders. ..................................................................................... 29
Figure 3-24: VM settings. Storage. ............................................................................................... 30
Figure 3-25: VM settings. Audio.................................................................................................... 30
Figure 3-26: VM settings. Network. ............................................................................................... 31
Figure 3-27: VM settings. Serial Ports. .......................................................................................... 31
Figure 3-28: VM settings. USB...................................................................................................... 32
Figure 3-29: VM settings. User Interface. ...................................................................................... 32
Figure 3-30: Start VM. ................................................................................................................ 33
Figure 3-31: Start VM. Warning. ................................................................................................... 33
Figure 3-32: Start VM. Ubuntu. .................................................................................................... 34
Figure 3-33: Start VM. Ubuntu. .................................................................................................... 34
Figure 3-34: If the VM does not resize to the monitor, you need to update the Guest Additions. ........... 35
Figure 3-35: Ubuntu. Update Guest Additions. ................................................................................ 35
Figure 3-39: Ubuntu. Resize the VM. ............................................................................................. 37
Figure 3-40: Ubuntu. Resize the VM. ............................................................................................. 38
Figure 3-41: Shared folder in Windows. ......................................................................................... 39
Figure 3-42: Shared folder settings in the VM................................................................................. 39
Figure 3-43: Shared folder settings in the VM. Need to update the Path in Windows............................ 40
Figure 3-44: Shared folder settings in the VM. Need to update the Path in Windows............................ 40
Figure 3-45: Update share folder. ................................................................................................. 41
Figure 3-46: Update shared folder. ............................................................................................... 42
Figure 3-47: Unmounted share folder. ........................................................................................... 42
Figure 3-48: Mounted share folder. ............................................................................................... 43
Figure 3-49: Script to mount the shared folder: mount_shared_folder.sh. ......................................... 43
Figure 3-54: Ubuntu software updater. .......................................................................................... 45
Figure 3-55: Updating Ubuntu with apt.......................................................................................... 45
Figure 3-56: Ubuntu Applications .................................................................................................. 46
Figure 3-57: Find Dolphin ............................................................................................................ 46
Figure 3-58: Dolphin file explorer ................................................................................................. 47
Figure 3-59: Notepadqq .............................................................................................................. 48
Figure 3-60: Open a file in Notepadqq through the file explorer ........................................................ 49

6 of 115
Date: 14/09/2020
Subject EODP
Figure 3-61: Notepadqq .............................................................................................................. 49

Figure 3-62: Download Panoply .................................................................................................... 50
Figure 3-63: Panoply script .......................................................................................................... 51
Figure 3-64: Launch Panoply ........................................................................................................ 51
Figure 3-65: Launch Panoply ........................................................................................................ 51
Figure 3-66: Panoply welcome window .......................................................................................... 52
Figure 3-67: Panoply open file ...................................................................................................... 53
Figure 3-68: Panoply. Double click on variable to open viewer. ......................................................... 54
Figure 3-69: Panoply. Variable viewer ........................................................................................... 55
Figure 3-70: Panoply. Variable array. Change the format for a higher precision as the default is %.1f ... 56
Figure 3-70: VM initial size, 69 GB ................................................................................................ 57
Figure 3-70: Shrinking the size of the VM ...................................................................................... 57
Figure 3-70: Run PowerShell as admin .......................................................................................... 58
Figure 3-70: Shrinking the VM size with Windows PowerShell ........................................................... 58
Figure 3-70: Shrinking the VM size with Windows PowerShell ........................................................... 59
Figure 3-70: VM size after shrinking .............................................................................................. 59
Figure 4-1: Open a terminal ......................................................................................................... 60
Figure 4-2: Open new tabs in the same terminal window or a new terminal ....................................... 61
Figure 4-3: The current and above directories ................................................................................ 62
Figure 4-4: Example of cd, ll and pwd ........................................................................................... 62
Figure 4-5: Double tab provides the options. In this example, when we type l1, there are two options
because there are two folders named l1b and l1c ............................................................................ 63
Figure 4-6: Reverse search. ......................................................................................................... 64
Figure 4-7: Reverse search. ......................................................................................................... 64
Figure 4-8: Editing a file in the terminal with nano. ......................................................................... 64
Figure 4-9: Example of permissions. Here I’m removing the eodp repository. There are git files in the .git
folder owned by root, so it ‘complains’ when I try to remove them. ................................................... 65
Figure 4-10: By using sudo (superuser), the remove is done without any warning. ............................. 65
Figure 4-11: Example of mkdir and ‘rm -r’. .................................................................................... 66
Figure 4-12: Example of mv. In this example my_folder1 is moved into my_folder2 ........................... 66
Figure 4-13: Size of a file with du. ................................................................................................ 67
Figure 4-14: Size of a directory with du. ........................................................................................ 67
Figure 4-15: Example of compressing with tar. Verbose is enabled (the ‘v’), so all the files included in the
TGZ are displayed. ...................................................................................................................... 68
Figure 4-16: Example of compressing with tar. ............................................................................... 68
Figure 4-17: Example of uncompressing with tar. ........................................................................... 68
Figure 4-18: Example of a symbolic link. ....................................................................................... 68
Figure 4-19: Example of use of an asterisk (*) ............................................................................... 69
Figure 4-20: Example of use of grep ............................................................................................. 69
Figure 4-21: Example of use of grep, case insensitive ..................................................................... 69

7 of 115
Date: 14/09/2020
Subject EODP
Figure 4-22: Example of locate ..................................................................................................... 70

Figure 4-23: Example of find ........................................................................................................ 70
Figure 5-1: Reference source code. The ISM/L1B/L1C modules have the same structure and are invoked
with the same call....................................................................................................................... 72
Figure 5-2: Reference source code (not including the modules). Contains the auxiliary data files, common
Python source code, and the configuration of the ISM/L1B/L1C. ....................................................... 73
Figure 5-3: Snapshot of the global and ISM local configuration files. ................................................. 74
Figure 5-4: Call to the ISM Module. ............................................................................................... 75
Figure 5-5: Saving the log messages with tee ................................................................................ 79
Figure 5-6: Saving the log messages with tee ................................................................................ 79
Figure 6-1: Create a GitHub account ............................................................................................. 80
Figure 6-2: Git homepage ............................................................................................................ 81
Figure 6-3: Git new repository ...................................................................................................... 82
Figure 6-4: Git new repository ...................................................................................................... 83
Figure 6-5: Git create develop branch ........................................................................................... 83
Figure 6-6: Git develop branch ..................................................................................................... 84
Figure 6-7: change directory to where you want to download the repository ...................................... 84
Figure 6-8: Git repository URL ...................................................................................................... 85
Figure 6-9: Git clone repository .................................................................................................... 85
Figure 6-10: Git clone repository .................................................................................................. 85
Figure 6-11: Git status ................................................................................................................ 86
Figure 6-13: Git switch to the develop branch ................................................................................ 86
Figure 6-14: Opening the README file with Notepadqq .................................................................... 87
Figure 6-15: Editing the README.md file in Notepadqq.................................................................... 87
Figure 6-17: Git add file .............................................................................................................. 88
Figure 6-18: Git commit .............................................................................................................. 88
Figure 6-19: Initial README.md in the origin/develop branch ........................................................... 88
Figure 6-20: Git commit history .................................................................................................... 88
Figure 6-21: Git log history – the changes to the README.md are in the local repository but not in the
origin (GitHub server) yet. ........................................................................................................... 89
Figure 6-22: Git push changes from the local/develop to the origin/develop branch. ........................... 89
Figure 6-23: README.md in the origin/develop branch after pushing changes .................................... 89
Figure 6-24: Git commit history after pushing changes .................................................................... 90
Figure 6-25: Sources of the project. .............................................................................................. 91
Figure 6-26: Download zip (this is directly in the VM with Firefox). ................................................... 91
Figure 6-27: Save the zip ............................................................................................................ 92

8 of 115
Date: 14/09/2020
Subject EODP
Figure 6-30: Unzip ...................................................................................................................... 94

Figure 6-31: Archive Manager ...................................................................................................... 95
Figure 6-32: Archive Manager ...................................................................................................... 96
Figure 6-33: Destination folder, your repository folder .................................................................... 97
Figure 6-34: Skip overwritting the README.md .............................................................................. 98
Figure 6-35: Replace the .gitignore file .......................................................................................... 99
Figure 6-36: Sources extraction complete .................................................................................... 100
Figure 6-37: Once completed, delete the zip file (or move it somewhere else). ................................. 101
Figure 6-38: Untracked files. ...................................................................................................... 102
Figure 6-39: Sources added. ...................................................................................................... 103
Figure 6-40: Commit. ................................................................................................................ 103
Figure 6-41: Push to origin/develop. ........................................................................................... 103
Figure 6-42: Sources are now in origin/develop. ........................................................................... 104
Figure 6-43: Git develop and master branches. You will commit to origin/develop. (Optional) you can
push a stable version to master. Source: nvie.com ....................................................................... 105
Figure 6-44: Git create pull request ............................................................................................ 105
Figure 6-45: Git create pull request ............................................................................................ 106
Figure 6-46: Git merge pull request ............................................................................................ 107
Figure 6-47: Git merge pull request ............................................................................................ 107
Figure 6-48: Git develop successfully merged into master.............................................................. 108
Figure 6-49: Git develop and master branches. You will commit to origin/develop. (Optional) you can
push a stable version to master. Source: nvie.com ....................................................................... 109
Figure 6-50: Example of “pulling” the origin/develop into the local feature branch. ........................... 110
Figure 6-51: A pull request. In this example I’m requesting to merge feature/FLEXFIPSGP-33 branch into
develop. .................................................................................................................................. 111
Figure 6-52: Git master, develop and “feature” branches. Each developer works in its feature branch.
They will pull the origin/develop branch to their local branch to fetch the latest changes. Afterwards the
user will commit his/her changes to his/her branch, push it to origin/feature branch, and request a pull
request. A different user will approve this request and the changes shall be merged to develop. Source:
researchgate.net ...................................................................................................................... 112
Figure 6-53: Export the source code in GitHub. ............................................................................ 113

9 of 115
Date: 14/09/2020
Subject EODP
1. INTRODUCTION
This document describes the development environment of Earth Observation Data Processing (EODP).
A Virtual Machine (VM) is provided, to have a common environment for all the participants of EODP.
Instructions are provided on how to install and start working with the VM in § 3.
The guest Operating System (OS) in the VM is Ubuntu. Ubuntu is an easy graphical Linux distribution.
Some basic commands are explained in § 4 (which are common for all Unix based systems).
Section § 5 describes the reference source code, and how to execute it. The EODP code is in two
languages: the Geometry Module is implemented and provided in C++, and the Instrument Module, L1B
and L1C modules are implemented in Python.
Finally, in § 6 the version control that will be used in EODP, Git through GitHub, is described.

10 of 115
Date: 14/09/2020
Subject EODP
2. REFERENCES
2.1. Applicable Documents

Table 2-1: Applicable Documents
Ref. Code Title Issue

[EODP_GUI] - EODP Guide to the Subject 1.0
[EODP_PY] - EODP Quickstart to Python for Matlab Programmers 1.0
2.2. Reference Documents

Table 2-2: Reference Documents
Ref. Code Title Issue

[RD01]
2.3. Acronyms
Table 2-3: Acronyms
Acronym Description
AD Applicable Document
ATBD Algorithm Theoretical Baseline Document
COTS Commercial Off The Shelf
EODP Earth Observation Data Processing
FYI For Your Information
GM Geometry Module
ISM Instrument Module
L1B Level-1B Processing Module
L1C Level-1C Processing Module
OS Operating System
RD Reference Document
TBC To be confirmed
TBD To be defined
TBW To be written
TN Technical Note
UT Unitary Test
VM Virtual Machine

11 of 115
Date: 14/09/2020
Subject EODP
3. SETTING UP THE WORK ENVIRONMENT.

This section provides the guide to set up the Virtual Machine (VM), which contains all of the software
needed. This is a furnished item, see [EODP_GUI], section 5.4 “Virtual Machine”.
NOTE: In the case that your base OS is some Linux distribution, you can directly work there instead of
importing the VM. However, this is at your own risk. One library that might be problematic is the NetCDF
and its base HDF5 for C++ because it uses different OS-native low-level libraries which are incompatible
between different distributions. The rest should be ~fine. For the preliminary list of COTS needed, see
[EODP_GUI] section 5.4.2 “COTS”.
3.1. Install Oracle VM Virtualbox

The first step is to install VM Virtualbox.
NOTE: if you have another tool to work with Virtual Machines, for example VMware, you can use that
instead.
Go to the Oracle download page https://www.virtualbox.org/wiki/Downloads and download the version
for the host Operating System (OS). Host OS whatever OS you have installed in your laptop. The next
steps will be done for a Windows OS.
Figure 3-1: Download page for VM Virtualbox
Click and save the executable somewhere:

12 of 115
Date: 14/09/2020
Subject EODP
Figure 3-2: Save executable for VM Virtualbox
The installation is straightforward, just press Next all the time. Screenshots are provided here as a
guideline:
Figure 3-3: Installation of VM Virtualbox

13 of 115
Date: 14/09/2020
Subject EODP

14 of 115
Date: 14/09/2020
Subject EODP

15 of 115
Date: 14/09/2020
Subject EODP
Finally, open Virtualbox (double-click on the newly created Desktop icon):

16 of 115
Date: 14/09/2020
Subject EODP
Figure 3-10: VM Virtualbox
3.2. Import the Virtual Machine

You should have available the Virtual Machine provided with the subject, a .ova file. See [EODP_GUI],
section 5.4 “Virtual Machine” for details.
Click on “Import Appliance”:

17 of 115
Date: 14/09/2020
Subject EODP
Figure 3-11: VM Virtualbox. Import Appliance.
This opens a menu where you can navigate to the location of the OVA file:

18 of 115
Date: 14/09/2020
Subject EODP

19 of 115
Date: 14/09/2020
Subject EODP

20 of 115
Date: 14/09/2020
Subject EODP
Modify, if you want, the destination folder for your VM:

21 of 115
Date: 14/09/2020
Subject EODP
Figure 3-15: VM Virtualbox. Destination folder for the VM.
Click on “Import” which launches the process:
Figure 3-16: VM Virtualbox. Import.

22 of 115
Date: 14/09/2020
Subject EODP
Once the import is finished, the Virtual Machine appears in the main panel of VM Virtualbox:
Figure 3-17: VM Virtualbox. Import.
3.3. Settings
You probably will not have to change the settings, but if you need to, here are several indications. First
right-click on the VM and select ‘Setting’s:

23 of 115
Date: 14/09/2020
Subject EODP
Figure 3-18: VM settings.
Make sure the clipboard is ‘Bidirectional’. This means that any text (like an URL) that you copy in the host
OS you can past in the VM, or vice versa.
NOTE: the drag-and-drop option is set to ‘Bidirectional’ but we will use another way to access files with a
shared folder, see § 3.6.

24 of 115
Date: 14/09/2020
Subject EODP
Figure 3-19: VM settings. General.
The System settings is an important configuration. The Base Memory is how much RAM you allocate to
the VM. The recommendation is to maximise it to the green area. The read area is how much your hows
OS needs (e.g., how much RAM Windows needs). In my laptop I have 32GB, which is more than needed,
so I have allocated 11GB to the VM. Basically, slide the pointer to the end of the green area.

25 of 115
Date: 14/09/2020
Subject EODP
Figure 3-20: VM settings. System.
The Display settings should be OK, make sure you have maximum video memory and you have enabled
3D acceleration. Number of monitors is one (one screen).

26 of 115
Date: 14/09/2020
Subject EODP
Figure 3-21: VM settings. System.
The shared folder is an important setting, as this is the folder that will be used to transfer files
between the host OS and the VM.
❑ First, create a folder in your host OS (e.g., in this example its called VBox_shared_folder in my user).
❑ You need to modify this shared folder, so double click on it, see Figure 3-23. Update path and name.\
❑ Do not select auto-mount or read-only (its better to manually mount it, see § 3.6). Follow the
configuration in Figure 3-23.
❑ If you want, you can change the name of the shared folder in Linux “my_shared_folder”. Or leave it
as it is.

27 of 115
Date: 14/09/2020
Subject EODP
Figure 3-22: VM settings. Shared Folders.

28 of 115
Date: 14/09/2020
Subject EODP
Figure 3-23: VM settings. Shared Folders.
The rest of the settings should be left by default. They are screenshot here for reference:

29 of 115
Date: 14/09/2020
Subject EODP
Figure 3-24: VM settings. Storage.
Figure 3-25: VM settings. Audio.

30 of 115
Date: 14/09/2020
Subject EODP
Figure 3-26: VM settings. Network.
Figure 3-27: VM settings. Serial Ports.

31 of 115
Date: 14/09/2020
Subject EODP
Figure 3-28: VM settings. USB.
Figure 3-29: VM settings. User Interface.
3.4. Run the Virtual Machine

Open the VM by either double-clicking the VM or with the green start arrow:

32 of 115
Date: 14/09/2020
Subject EODP
Figure 3-30: Start VM.
You might get a warning, just click OK:
Figure 3-31: Start VM. Warning.
Then you will see the Ubuntu welcome window:

33 of 115
Date: 14/09/2020
Subject EODP
Figure 3-32: Start VM. Ubuntu.
After that, the VM should open.
Figure 3-33: Start VM. Ubuntu.
If your VM does not maximise to the full screen, go to the next section “Guest Additions”.
3.5. Guest Additions.

Your VM should resize to the size of your monitor. If it does not, you would need to update the Guest
Additions.

34 of 115
Date: 14/09/2020
Subject EODP
Figure 3-34: If the VM does not resize to the monitor, you need to update the Guest Additions.
Figure 3-35: Ubuntu. Update Guest Additions.

35 of 115
Date: 14/09/2020
Subject EODP
You are prompted for the superuser password, which you can find in [EODP_GUI] section 5.4. “Superuser
Credentials”.

36 of 115
Date: 14/09/2020
Subject EODP
After that, the installation is complete. However, there is one last step, which is to resize to whatever
your screen resolution is:
Figure 3-39: Ubuntu. Resize the VM.

37 of 115
Date: 14/09/2020
Subject EODP
Now you should get the VM full screen. It should also automatically resize if you change screen or if you
modify the VM size.
Figure 3-40: Ubuntu. Resize the VM.
3.6. Shared folder. Accessing the files in the VM from the host OS.
This section details how to create a shared folder, i.e., a folder in your host OS (e.g., Windows) that is
visible in the VM. A shared folder is useful to:
❑ move the reference inputs and source code to the VM
❑ move the outputs generated in the VM to the host OS
❑ to store big files without increasing the size of the VM (see § 3.10).
3.6.1. Create a folder in the host OS

The first step is to create a folder in the host OS. Also review the settings for the Shared folder, Figure
3-23. In the example in § 3.3 there are two linked folder:
❑ host OS (Windows): C:\Users\34655\VM_shared_folder

38 of 115
Date: 14/09/2020
Subject EODP
Figure 3-41: Shared folder in Windows.
3.6.2. Update the shared folder settings in the VM
Figure 3-42: Shared folder settings in the VM.
As you can see, the path in Windows of the shared folder does not match the shared folder I have
created in Figure 3-41, so I need to update it:

39 of 115
Date: 14/09/2020
Subject EODP
Figure 3-43: Shared folder settings in the VM. Need to update the Path in Windows.
Double click on the shared folder and the navigate to ‘Other’ to select your shared folder in the host OS:
Figure 3-44: Shared folder settings in the VM. Need to update the Path in Windows.

40 of 115
Date: 14/09/2020
Subject EODP
Figure 3-45: Update share folder.
Click OK. Leave Auto-mount and Read-only unchecked:

41 of 115
Date: 14/09/2020
Subject EODP
Figure 3-46: Update shared folder.
3.6.3. Mounting the shared folder

From the steps above we have defined a shared folder in the VM called “my_shared_folder”. It should be
located in the home, so:
/home/luss/my_shared_folder
If you have changed the name, remember to create a directory with the name you have selected (mkdir
§ 4.2.3.2)
If you navigate there and see the contents, you will not see the actual contents of the shared folder:
Figure 3-47: Unmounted share folder.
We are going to manually mount the shared folder:

sudo mount -t vboxsf VM_shared_folder my_shared_folder/
Now if you check the contents of the shared folder you will see the same contents than in Figure 3-41:

42 of 115
Date: 14/09/2020
Subject EODP
Figure 3-48: Mounted share folder.
3.6.4. Mounting the shared folder - script

TIP: You need to mount the VM every time you open the VM, so you can save the command above in a
small script and simply execute it. You can find in the VM already a script for this, in case you want to
use it:
Figure 3-49: Script to mount the shared folder: mount_shared_folder.sh.
NOTE: if you create your own script, remember to give it execution permissions (755)
chmod 755 script
You might need to modify if you have changed the name of the folder in the host OS or the VM.
Run the script (with a “./”):
It will prompt you for superuser credentials that you can find in [EODP_GUI] section 5.4. “Superuser
Credentials”. After that, the folder is mounted.
3.7. Close the VM

To close the VM click on File>Close:

43 of 115
Date: 14/09/2020
Subject EODP
Select any of the three options:

❑ Save the machinestate: is the equivalent of Sleep. Keeps a pointer of the state and everything you
have open.
❑ Shutdown: closes all of the programs and shuts down (same than in Windows).
❑ Power off is the equivalent of force-closing everything. In general it’s better to shutdown.
3.8. Ubuntu updates

Every so often, when you log in you will get a pop-up from Ubuntu offering to update the system:

44 of 115
Date: 14/09/2020
Subject EODP
Figure 3-54: Ubuntu software updater.
The example above is a massive update, in general it should only be a couple of MBs. Click on “Install
now”.
You can also update your system via command line with:
sudo apt update
This will parse the system and identify which packets need to be updated.
sudo apt upgrade
This will upgrade the packets, so actually install the packets.
Sudo executes a command as root (“superuser do”), so you will be prompted for the administrator
password, see [EODP_GUI] section 5.4. “Superuser Credentials”.
Figure 3-55: Updating Ubuntu with apt.
3.9. Test COTS installed in the VM

Now that the VM is running, test a couple of basic COTS (“Commercial off the shelf” software) installed in
the Virtual Machine.

45 of 115
Date: 14/09/2020
Subject EODP
3.9.1. Dolphin (file explorer)

Dolphin is a file explorer for Ubuntu (here are other as well). To open it, click on the bottom left icon of 9
dots “Applications”:
Figure 3-56: Ubuntu Applications
In the search start typing ‘Dolphin’ until the Dolphin icon appears, and double-click to open it:
Figure 3-57: Find Dolphin
This will open the Dolphin window. You can move to any directory and do the typical operations of any
file explorer.

46 of 115
Date: 14/09/2020
Subject EODP
Figure 3-58: Dolphin file explorer
3.9.2. Notepadqq (text editor)

There are many ways to open and edit files in Linux. You can use the vim editor or nano through the
command line, or for example gedit or NEdit which open windows. In the VM I have included Notepadqq,
which is quite similar to Windows’ Notepad++. I have included the desktop icon to the left, so you only
need to double-click on it to open the editor.

47 of 115
Date: 14/09/2020
Subject EODP
Figure 3-59: Notepadqq
You can also open any file through the file explorer by right-clicking on the file and selecting ‘Open with’:

48 of 115
Date: 14/09/2020
Subject EODP
Figure 3-60: Open a file in Notepadqq through the file explorer
Figure 3-61: Notepadqq
3.9.3. Panoply (NetCDF viewer)

Panoply is a tool created by NASA to view NetCDF files. Panoply is already installed in the VM, but in case
you want to have it also in your host OS, it is available for macOS, Linux and Windows in here:
https://www.giss.nasa.gov/tools/panoply/download/

49 of 115
Date: 14/09/2020
Subject EODP
Figure 3-62: Download Panoply
To launch it you need to run the shell script located in the Panoply folder:

50 of 115
Date: 14/09/2020
Subject EODP
Figure 3-63: Panoply script
You can run it from any place using a ‘./’. Add an ‘&’ so that you can continue using the window:
Figure 3-64: Launch Panoply
And running Panoply from a different folder:
Figure 3-65: Launch Panoply
This launches Panoply:

51 of 115
Date: 14/09/2020
Subject EODP
Figure 3-66: Panoply welcome window
Panoply is intuitive to use. You select any NetCDF and open it. Then you can inspect a variable by double-
clicking on it. Another window will open with a viewer for the image, Figure 3-69. You can also inspect
the values of the variable, see Figure 3-70.

52 of 115
Date: 14/09/2020
Subject EODP
Figure 3-67: Panoply open file

53 of 115
Date: 14/09/2020
Subject EODP
Figure 3-68: Panoply. Double click on variable to open viewer.

54 of 115
Date: 14/09/2020
Subject EODP
Figure 3-69: Panoply. Variable viewer

55 of 115
Date: 14/09/2020
Subject EODP
Figure 3-70: Panoply. Variable array. Change the format for a higher precision as the default is %.1f
3.10. Tips for working with a VM.

Several random comments regarding the VM:
❑ Keep the VM as minimal as possible. Keep it clean. Delete old files, remove programs you don’t use.
The leaner the VM, the faster it will run and the lower the probability of problems.
❑ The VM increases size. It is not “smart”, so it increases size with time. (You can reduce its size, but it
is not straightforward). The VM has dynamic memory allocation. For example, if you generate 1GB of
outputs, it increases its size in ~1GB. Once you delete them, it pads the memory with zeroes, but it
does not reduce its size. It keeps the size of the memory, that it will use later on. Therefore, it is
important to keep it as clean as possible. FYI, I keep my VM very minimal and after a while it weights
~50GB.
❑ Move outputs that you want to save to the shared folder and delete them from the VM.
Alternatively, save them directly in the shared folder. Basically, files in the shared folder are “held” in
the host OS, so they don’t increase the size of the VM.
❑ If you don’t have a lot of free space on your PC/laptop, you can locate and run the VM directly from
an external hard drive. It is recommended to use an USB 3.0 port, and it should work fine, without
a significant degradation in time performances.
❑ Shutdown the VM after you finish working with it because Windows tends to crash when the VM is left
open :)

56 of 115
Date: 14/09/2020
Subject EODP
3.11. Shrinking the size of the VM

NOTE: this section is oriented to Windows Host Operating System.
Even taking all the precautions listed in § 3.10, cleaning up the Virtual Machine (VM) regularly, etc., you
might see that the VM slowly starts growing in size. This is because the VM allocates memory as it needs
it, and even if you delete the files from the VM, it does not deallocate the space. Here are the steps to
shrink the size of the VM, this will deallocate all the space not currently in use.
Figure 3-71: VM initial size, 69 GB
❑ Run the VM, open a terminal, and type (it will prompt you for the superuser password):
sudo dd if=/dev/zero of=/var/tmp/bigemptyfile bs=4096k ; sudo rm /var/tmp/bigemptyfile
Figure 3-72: Shrinking the size of the VM
❑ Close the VM
❑ In Windows, run the PowerShell as Admin:

57 of 115
Date: 14/09/2020
Subject EODP
Figure 3-73: Run PowerShell as admin
❑ In the PowerShell console, change directory to the VirtualBox folder:

cd 'C:\Program Files\Oracle\VirtualBox\'
Figure 3-74: Shrinking the VM size with Windows PowerShell
❑ Now, run the following command in the Windows PowerShell, substituting the location and name of
the VDI file:
.\VBoxManage.exe modifymedium --compact 'C:\Users\34655\VirtualBox
VMs\EODP_VM\FLEX_VM_Ubuntu_18.04_LTS-disk001.vdi'

58 of 115
Date: 14/09/2020
Subject EODP
You will see the progress:
Figure 3-75: Shrinking the VM size with Windows PowerShell
And that is it. See how the VM size is reduced by more than 30GB:
Figure 3-76: VM size after shrinking

59 of 115
Date: 14/09/2020
Subject EODP
4. COMMAND PROMPT OR TERMINAL.

Ubuntu does have a lot of Graphical User Interface, but the nominal way of working in Linux is through
the command prompt or terminal. In this section the basic, most used commands are detailed.
4.1. Open a window and new tabs

Opening a terminal window can be done simply by double-clicking the icon on the left of the Virtual
Machine:
Figure 4-1: Open a terminal
You can add new tabs or open more than one window:

60 of 115
Date: 14/09/2020
Subject EODP
Figure 4-2: Open new tabs in the same terminal window or a new terminal
4.2. Some basic operations

In this section, some of the most common commands are detailed. If you have any doubt of what a
command does, google it first! Never type a command in the terminal that you do not know
what it does. It can get you in a lot of trouble
4.2.1. Change directory and see contents
4.2.1.1. ‘cd’ to a directory, ‘ll’ to list contents

The ‘ll’ command lists the contents of a directory.
ll Lists contents of current directory.
ll my_directory Lists contents of directory ‘my_directory’.
cd is used to change directory.

cd my_directory Changes directory to ‘my_directory’.
cd Changes directory to home (/home/luss/)
cd - Changes directory to the previous directory you were in
4.2.1.2. The ‘.’ and ‘..’ directories

The current directory is ‘.’. The directory above is ‘..’.
So, for example, ‘ll .’ will list the files in the current directory, and ‘ll ..’ the files in the directory above.

61 of 115
Date: 14/09/2020
Subject EODP
Figure 4-3: The current and above directories
4.2.1.3. ‘pwd’ see which directory you are in

The pwd command tells you where you are.
Figure 4-4: Example of cd, ll and pwd

62 of 115
Date: 14/09/2020
Subject EODP
4.2.2. Making it easier to input a command
4.2.2.1. Autocomplete with Tab

You do not need to finish writing the full name of a directory, or the full command. Linux will help you by
autocompleting it by using the Tab keyboard key.
In fact, it is safer to autocomplete. If you press tab and it doesn’t autocomplete, the input path is not
correct.
In case there is more than one option, double-tab will show you the available options.
Figure 4-5: Double tab provides the options. In this example, when we type l1, there are two options
because there are two folders named l1b and l1c
4.2.2.2. Arrows Up and Down for commands

If you want to re-execute a previous command, just press the top arrow. The top and bottom arrows
allow you to navigate the command history.
4.2.2.3. Reverse search Ctrl+R

If you have typed a long command a long while ago that you want to reuse, a useful command is the
reverse search. With the Control Key (Ctrl) plus the letter R (Ctrl+R) you start the reverse search. Start
typing your query and the terminal will show suggestions. When you find the one you want, use Tab to
complete it.

63 of 115
Date: 14/09/2020
Subject EODP
An example. I want to retrieve a commit to avoid having to type all the ‘git commit -m “FLEXFIPSGP-33:
blabla”’. I press Ctrl+R and I start typing ‘git’. The first result I get is this, a git pull, which is not what
I’m interested in.
Figure 4-6: Reverse search.
I continue typing ‘git c’ and when I get there I already have a suggestion of a commit command. I then
press Tab to finish.
Figure 4-7: Reverse search.
4.2.3. Files and Directories
4.2.3.1. Create a file (nano)

To create a file there are several ways, using Notepadqq (graphical), or via the terminal. If you want to
create it in the terminal, one option is to use nano. ‘nano’ is a terminal editor. You can use it to create,
open and edit files.
nano my_file
Modify the file and then click Ctrl+S to save and Ctrl+X to close (the commands are listed in the
bottom.
NOTE: You need to have write permissions in the folder where you are trying to save and edit the file.
See chmod below.
Figure 4-8: Editing a file in the terminal with nano.
4.2.3.2. Create a directory (mkdir)

Create a directory using mkdir. If you want to create a directory called ‘my_directory’, you would type:

64 of 115
Date: 14/09/2020
Subject EODP
mkdir my_directory
NOTE: You need to have write permissions in that folder. See chmod below.
4.2.3.3. Remove a file (rm)

To remove a file use rm.
rm my_file
If you try to remove files that have been created by root (superuser) you will get an error. In this case
you need to run rm with sudo.
sudo rm my_file
Below is an example where I delete the repository eodp/ (thanks to GitHub there’s no problem). The
hidden .git folder has files that are protected and it warns me against deleting them.
Figure 4-9: Example of permissions. Here I’m removing the eodp repository. There are git files in the .git
folder owned by root, so it ‘complains’ when I try to remove them.
If you use sudo, you do not have this issue.
Figure 4-10: By using sudo (superuser), the remove is done without any warning.
Be very careful doing sudo rm *. There is no undoing this operation.
4.2.3.4. Remove a directory (rm -r)

To remove a directory use rm -r. The ‘-r’ stands for recursive.
rm -r my_directory
4.2.3.5. Copy a file or a directory (cp, cp -r)

To copy a file use cp
cp my_file my_file_newname
To copy a directory use cp -r
cp -r my_directory my_directory_newname
4.2.3.6. Move a file or a directory (mv)

The mv command moves a file or a directory
mv my_directory destination

65 of 115
Date: 14/09/2020
Subject EODP
Figure 4-11: Example of mkdir and ‘rm -r’.
Figure 4-12: Example of mv. In this example my_folder1 is moved into my_folder2
4.2.4. Change permissions (chmod)

Occasionally you need to change permissions to a folder or file. One example is if you create a script, you
need to give it execution permissions. In general be careful what you do with chmod.

66 of 115
Date: 14/09/2020
Subject EODP
First, to give read+execute permissions to a file do:

chmod 755 my_file
To give read+execute permissions to a directory do:
chmod -R 755 my_folder
To give full permissions (read, write & execute) do:
chmod 755 my_file
chmod -R 777 my_folder
4.2.5. See how much a directory or file weighs (du -hs)

The basic command to see how much a file weighs is du:
du my_file
The -h option translates the size into a human-friend (i.e., MB, GB, etc.) size.
du -h my_file
Figure 4-13: Size of a file with du.
For directories, use the -s option. It provides the summary of the size, the total of the directory (if not
used, it will printout the file for each file inside the directory).
du -hs my_directory
Figure 4-14: Size of a directory with du.
4.2.6. Compress or uncompress with tar

A TFZ file is a TAR file compressed using GNU zip software. To compress a file or directory, this is the
command:
tar -czvf mytgz.tgz my_directory
To uncompress a TGZ do:
tar -xzvf mytgz.tgz

67 of 115
Date: 14/09/2020
Subject EODP
Figure 4-15: Example of compressing with tar. Verbose is enabled (the ‘v’), so all the files included in the
TGZ are displayed.
Figure 4-16: Example of compressing with tar.
Figure 4-17: Example of uncompressing with tar.
4.2.7. Symbolic links

A symbolic link in Linux is like a shortcut in Windows. To create a symbolic link, the syntax is the
following:
ln -s origin destination
In the following example I create a symbolic link for a binary located in the ut02 folder of the gm test, in
the current directory (‘.’). When you do an ‘ll’, you see the symbolic link in cyan with an arrow showing
where that link is redirected. If I want to execute the binary, I only need to do ‘./FLEX-FIPS-GM_UT02’
and saves me from having to input the whole path.
Figure 4-18: Example of a symbolic link.
4.2.8. Run a script or an executable

You can run a shell script or an executable from the command line. The syntax is to place a ‘./’ plus the
executable (called the “binary”) or script (.sh file).
./binary

68 of 115
Date: 14/09/2020
Subject EODP
NOTE: there is a lot of context missing here as executables and scripting is a whole topic. It is out of
scope for this subject.
An example is launching Panoply, see § 3.9.3.
4.2.9. Asterisk (*)

In Linux, the asterisk (*) is a wildcard, i.e. a placeholder for one or any number of unspecified
characters. It can be used anywhere.
Figure 4-19: Example of use of an asterisk (*)
4.2.10. Searching
4.2.10.1. Find in file (grep)

grep is the basic command to look for a string inside a file. The basic syntax is:
grep -r “string” search_directory
❑ ‘-r’ is to search recursively in that directory

❑ ‘search_directory’ is the directory where you want to look in. It usually is ‘.’, the current directory.
❑ “string” is whatever string you are looking for. It is case sensitive. If you want to make it case
insensitive you have to use -i.
grep -ir “string” .
Figure 4-20: Example of use of grep
Figure 4-21: Example of use of grep, case insensitive
When you want to look only in a certain type of files, for example the C/C++ source or header files, use:
grep -r “string” --include=*.{cpp,hpp} .
Or for Python scripts:
grep -r “string” --include=*.py .

69 of 115
Date: 14/09/2020
Subject EODP
4.2.10.2. Find a file or folder (locate, find)

If you want to search for a file you can use locate or find.
Locate is a “brute force” search which looks anywhere in the system.
locate string
Figure 4-22: Example of locate
The command ‘find’ allows more control:

find search_directory -name “string”
Figure 4-23: Example of find

70 of 115
Date: 14/09/2020
Subject EODP
5. IMPLEMENTATION AND EXECUTION.
5.1. Python
For general support with the Python language, and starting with PyCharm (Python development
environment), please refer to [EODP_PY].
This section describes how to run the reference source code, which is the template on which you will
develop the algorithms of data processing.
5.1.1. The EODP source code in Python

The reference source code provided contains the main architecture of the software, the definition of the
classes and interfaces, plus several auxiliary functions.
The three modules that are implemented in Python are the Instrument (ISM), Level-1B (L1B) and
Level-1C (L1C) Modules.
First, this is the code architecture. There are three folders for each module: ism/, l1b/ and l1c/. Each of
those modules has a source folder src/, where the python classes are located. There is another folder,
test/, which is a place to add folders for each of the unitary tests (ut01/, ut02/, etc).
The folder for the gm/ is for the Geometry Module, is described in § 5.2. This module is implemented in
C++ and uses the EOCFIs.

71 of 115
Date: 14/09/2020
Subject EODP
Figure 5-1: Reference source code. The ISM/L1B/L1C modules have the same structure and are invoked
with the same call.
There are several other folders. The auxiliary/ folder contains input data, like constants, the log
configuration, the ISRF curves.
The common/ folder is a folder for generic Python functions used in the ISM, L1B and L1C. the
common/io/ folder contains folder to “input/output” information, so reading and writing. The
common/plot/ folder contains auxiliary functions to plot functions and images. Finally, the common/src/
folder has other generic functions.

72 of 115
Date: 14/09/2020
Subject EODP
Figure 5-2: Reference source code (not including the modules). Contains the auxiliary data files, common
Python source code, and the configuration of the ISM/L1B/L1C.
5.1.1.1. Configuration
The configuration files are stored in the config/ folder, there is one per module and one global
configuration file, see Figure 5-2.
The configuration files have the default parameters that define the system, the instrument and the
processing.

73 of 115
Date: 14/09/2020
Subject EODP
Figure 5-3: Snapshot of the global and ISM local configuration files.
5.1.1.2. Call
The call to the ISM, L1B and L1C follows the same logic. The modules are initialised with three
directories:
❑ auxdir: the path to the auxiliary/ directory
❑ indir: the path to the module inputs
❑ outdir: the path to the output directory, where you want to save the outputs.
Remember to import the ism class, as shown in the example in the figure below. You can find scripts to
launch the ISM, L1B and L1C standalone in the ism/, l1b/ and l1c/ folders.
The way to call the modules is:
from module.src.module import module
# Define auxdir, indir, outdir
myModule = module(auxdir, indir,outdir)
myModule.processModule()
Where ‘module’ is ‘ism’, ‘l1b’ or ‘l1c’.

74 of 115
Date: 14/09/2020
Subject EODP
Figure 5-4: Call to the ISM Module.
The code is structure in “classes” (pseudo-object oriented programming). Below is an example of the
class for the ISM. This class has two methods (i.e., functions), the class constructor __init__ and the
processModule(). The __init__ method is invoked when the class is created. So in Figure 5-4, in myIsm =
ism(auxdir, indir, outdir). The processModule is explicitely called in the following line.
The ISM calls three other classes, for the optical, detection and video phase. Notice how each class is
initialised and the called with the compute() method.
class ism(initIsm):
def __init__(self, auxdir, indir, outdir):

super().__init__(auxdir, indir, outdir)
def processModule(self):
# Read input TOA cube
sgm_toa, sgm_wv = readCube(self.indir, self.globalConfig.scene)
for band in self.globalConfig.bands:

self.logger.info("Start of BAND " + band)
# Optical Phase
myOpt = opticalPhase(self.auxdir, self.indir, self.outdir)
toa = myOpt.compute(sgm_toa, sgm_wv, band)
# Detection Stage
myDet = detectionPhase(self.auxdir, self.indir, self.outdir)
toa = myDet.compute(toa, band)
# Video Chain Phase

75 of 115
Date: 14/09/2020
Subject EODP
myVcu = videoChainPhase(self.auxdir, self.indir, self.outdir)

toa = myVcu.compute(toa, band)
# Write output TOA

writeToa(self.outdir, self.globalConfig.ism_toa + band, toa)
The compute method of these classes will have the calls to each of the functions. For example, here is
the sample code for the video chain phase:
def compute(self, toa, band):
# Electrons to Voltage - read-out & amplification

toa = self.electr2Volt(toa,
self.ismConfig.OCF,
self.ismConfig.ADC_gain)
# Digitisation
toa = self.digitisation(toa,
self.ismConfig.bit_depth,
self.ismConfig.min_voltage,
self.ismConfig.max_voltage)
return toa
In EODP, we are going to implement functions that we will add in the opticalPhase.py, detectionPhase.py
and videoChain.py, and we will include the call to these functions in the compute() methods, in order to
create a sequence of processing.
5.1.2. Python classes and “self”

Please follow the structure and syntax of the Python classes. Whenever you define a function inside a
class, the syntax is the following:
def myFun(self, var1, var2):
# do things
return toa
When you call this function in the class, follow this syntax:
toa = self.myFun(toa, var1, var2)
5.1.3. Saving the log in Python

Each execution run saves the log in the output folder for each module.
Additionally, you can also save the log by execute the script in the terminal and redirecting the output
with a tee command, see § 5.2.2.

76 of 115
Date: 14/09/2020
Subject EODP
python3 main.py 2>&1 | tee mylog.log
5.2. C++
5.2.1. The EODP Geometry Module in C++

The Geometry Module is implemented in C++ using the EOCFI libraries.
You can find the source code in
/home/luss/EODP/eodp_gm
5.2.1.1. Visualising the GM, the Eclipse IDE

In order to navigate through the code you can always use a text editor (Vim, nano, etc.). However, you
can use the Eclipse IDE (Integrated Development Environment). Eclipse helps you navigate through the
code with its many short cuts. For example press Ctrl+select a functions, and you navigate to that
function. I highly recommend that you learn to use Eclipse as you will probably have to use them in the
future (if you develop in Java, C, C++). Netbeans, the other main IDE, is very similar.
Figure 5-5: Eclipse IDE. Use the Project Explorer to access different files (left). On the right, you see all
the functions and attributes of the current class. To navigate, for example to initOrbitId(), press
Ctrl+mouse click the function.
5.2.1.2. Running the GM

The code available in the Virtual Machine is provided already compiled (see next section). You will find
the “binary” in the bin folder. That is the executable, “geometry_module”:

77 of 115
Date: 14/09/2020
Subject EODP
Figure 5-6: Eclipse IDE. Use the Project Explorer to access different files (left). On the right, you see all
the functions and attributes of the current class. To navigate, for example to initOrbitId(), press
Ctrl+mouse click the function.
You can run this binary with the following call. Go to the binary folder (you can also run it from a different
directory), and provide after the binary the location of the geometry local configuration file and an
output directory:
./geometry_module ../conf/gm_local_conf.xml output_dir
Figure 5-7: Executing the geometry module.
5.2.1.3. Compiling the code

In this subject, it is not required to program or modify the source code of the GM. However, I
wholeheartedly encourage you to experiment with C++ as a programming language, and the use of the
EOCFIs in particular.
Some possible extensions of the code that you can try are:
❑ Introduce errors to the orbit and attitude (a bias, or a Gaussian distribution).
❑ Try different attitude modes (geocentric pointing for example)
❑ Implement a different focal plane for each band
❑ Save the orbit/attitude in inertial frame.
❑ Save the attitude in Euler angles.
❑ [ADVANCED] Use the MKL library to interpolate the orbit, simulating the “Estimated orbit”, i.e., the
GNSS sensors. MKL is a math operating library from Intel which is very powerful. You will need to
install the MKL library, setting up the interpolation, etc. If you are interested in performances, code
optimisation, heavy data analyses you might want to try MKL (FFTs, convolutions, etc).
In a nutshell, after modifying your code in C/C++, you need to compile it, and then run it.
The compilation, explained in very simple terms, first, checks the syntax of your code, and then creates
an object (linking all the dependencies). With this “object” (the binary), you run the code.
You can setup Eclipse to compile the code there. However, I prefer to use the terminal directly to compile
and run. This project already has all the compilation options in the main project CMakeLists.txt (check it
out in the root folder of the project), so you will not need to modify it a priori (unless you want to! :) ).
To compile the code:
❑ Create a build directory in the root folder:
~/EODP/eodp_gm/build/

78 of 115
Date: 14/09/2020
Subject EODP
❑ In that build directory, you might need to do a cmake

cmake ..
❑ Finally, compile the code

make
NOTE: you can use all the cores you have assigned to your Virtual Machine, and it will compile faster:
make -j6 For 6 cores
make -j$nproc For the available amount of cores.
5.2.2. Saving the log in C++

When you execute something in the terminal, save the log using a tee. After the binary and its
arguments, add “2>&1 | tee” and the name of the log file you want.
binary arguments 2>&1 | tee mylog.txt
“2>&1” tells the system to redirect the standard output and error (what is printed in the command line),
and “| tee” duplicates the output so that it is printed in the screen and output to the file.
Figure 5-8: Saving the log messages with tee
After that, you see that a log file has been created:
Figure 5-9: Saving the log messages with tee

79 of 115
Date: 14/09/2020
Subject EODP
6. VERSION CONTROL WITH GIT
6.1. Create Account in GitHub

Go to https://github.com/join
Figure 6-1: Create a GitHub account
You might need to do some verification process.

There are some benefits for students, you can check them out here:
https://education.github.com/students
(We won’t need them, but in case you’re interested).
6.2. Create a Project in GitHub

Your Git homepage will look like this (minus the two repositories):

80 of 115
Date: 14/09/2020
Subject EODP
Figure 6-2: Git homepage
Click on the green button ‘New’ in order to create a new repository. Assign any name that you want,
add a description (optional). Remember to put it as Private and a README:

81 of 115
Date: 14/09/2020
Subject EODP
Figure 6-3: Git new repository
You have now created a new repository:

82 of 115
Date: 14/09/2020
Subject EODP
Figure 6-4: Git new repository
Notice how the repository is created with one branch, master. We are going to create another branch,
develop. We are going to commit to the develop branch (read about the typical workflow in § 6.7). Click
on the dropdown menu and type in ‘develop’:
Figure 6-5: Git create develop branch
Notice how the repository now has 2 branches:

83 of 115
Date: 14/09/2020
Subject EODP
Figure 6-6: Git develop branch
6.3. Clone your Project

To “clone” a project is to download it to local. First, open the terminal (§ 4.1) and change directory (‘cd’,
§ 4.2.1.1) to where you want to download the repository. Do an ll to check what is in the directory.
Figure 6-7: change directory to where you want to download the repository
Retrieve the repository URL from GitHub:

84 of 115
Date: 14/09/2020
Subject EODP
Figure 6-8: Git repository URL
Now, clone the repository with the following command:

git clone pathtorepo
Figure 6-9: Git clone repository
You will find the new repository. In this example it is a directory named ‘eodt_test2’, which contains a
.git/ directory and an empty README file:
Figure 6-10: Git clone repository
6.4. Git status

The git command used more often is
git status
Git status is used every other command, and it will tell you what the new files are, what files have been
modified and deleted.
Change directory to your repository and type git status:

85 of 115
Date: 14/09/2020
Subject EODP
Figure 6-11: Git status
First thing to notice, we are on the master branch. We will change that in a minute. Second thing, the
repository is up to date, meaning, we are aligned with the origin/master.
NOTE: ‘origin’ means the remote repository, i.e., the repository located at GitHub.
6.5. Git change branch to develop

Now we will change the branch to develop.
First, type:
git branch -a
You will get this:
The green branch, master, is the branch you have in your local. The other three branches in red are
located in origin, i.e., the GitHub server.
❑ origin/HEAD points at the main branch which by default is origin/master
❑ origin/master and origin/develop are the two branches we created in the repository, see Figure 6-6.
Now we are going to switch branch to develop with

git checkout -b develop
Do a git branch (or git status) to check that we are currently in develop:
Figure 6-13: Git switch to the develop branch
6.6. Git add and commit

Let’s do a fist change to the repo, before adding all of the sources.

86 of 115
Date: 14/09/2020
Subject EODP
6.6.1. Modify the README.md

Either open nano in the terminal (§ 4.2.3.1) or open the README file in Notepadqq (§ 3.9.2).
Add some text to the file, save and close:
Figure 6-14: Opening the README file with Notepadqq
Figure 6-15: Editing the README.md file in Notepadqq
NOTE: (optional) you can enhance the text with markdown. Info in
https://guides.github.com/features/mastering-markdown/.
Now do a git status:
Notice how the file is marked in red, meaning it has changed from the version in the repository.

87 of 115
Date: 14/09/2020
Subject EODP
Now we will add the file. This notifies Git that this file shall be included in the following commit. The
syntax is the following:
git add pathtofile
Figure 6-17: Git add file
Notice how the file is now marked in green. The following step is to commit it:
git commit -m “my commit message”
Figure 6-18: Git commit
Now head over to GitHub to see your changes. Open the README.md and check the commit history:
Figure 6-19: Initial README.md in the origin/develop branch
Figure 6-20: Git commit history
The changes we have done are not in the GitHub repository!! The reason for this is that the changes
are in our local repository. Do a git log to check out the commit history:

88 of 115
Date: 14/09/2020
Subject EODP
Figure 6-21: Git log history – the changes to the README.md are in the local repository but not in the
origin (GitHub server) yet.
We need to push the changes from our local develop branch to the origin/develop branch. This
means we need to upload our changes to the GitHub server. The syntax for this is:
git push origin develop
Figure 6-22: Git push changes from the local/develop to the origin/develop branch.
Now go back to your GitHub account and refresh the README.md and the commit history:
Figure 6-23: README.md in the origin/develop branch after pushing changes

89 of 115
Date: 14/09/2020
Subject EODP
Figure 6-24: Git commit history after pushing changes
6.6.2. Summary and tips

Summing up, the steps are
❑ modify something
❑ git add file(s)
❑ git commit -m “message”
❑ git push origin develop
Tips:
❑ Do a git status after every operation
❑ Do small, unitary commits. Do not wait to finish a whole algorithm to commit it. Do many
intermediate commits.
❑ Put meaningful commit messages that will help you in the future in case you need to trace older
versions.
❑ Remember to push to the origin! Assume that the VM can crash any minute (it should not but it is a
good practice).
6.6.3. Add the sources

Download the sources. Instructions and path in [EODP_GUI], section 5.2 “EODP Reference Code”.

90 of 115
Date: 14/09/2020
Subject EODP
Figure 6-25: Sources of the project.
You can directly download it in the VM through Firefox, or download it to the shared folder in the host OS
(e.g. Windows) and get it from there. Move the sources to the folder of your repository.
Figure 6-26: Download zip (this is directly in the VM with Firefox).

91 of 115
Date: 14/09/2020
Subject EODP
Figure 6-27: Save the zip
Navigate the the folder of your repository:

92 of 115
Date: 14/09/2020
Subject EODP
Unzip it by pressing right-mouse button ‘Archive Manager’:

93 of 115
Date: 14/09/2020
Subject EODP
Figure 6-30: Unzip

94 of 115
Date: 14/09/2020
Subject EODP
Figure 6-31: Archive Manager
Select all the files and click on Extract (top-left):

95 of 115
Date: 14/09/2020
Subject EODP
Figure 6-32: Archive Manager
This takes to you the destination folder. Select your repository folder, and click on the green button
‘Extract’:

96 of 115
Date: 14/09/2020
Subject EODP
Figure 6-33: Destination folder, your repository folder
It will ask you whether you want to overwrite or skip the README.md. Press Skip and keep your own
README.md.

97 of 115
Date: 14/09/2020
Subject EODP
Figure 6-34: Skip overwritting the README.md
It will prompt you again to overwrite the .gitignore. The .gitignore I provide is quite complete so I
recommend that you overwrite it with ‘Replace’:

98 of 115
Date: 14/09/2020
Subject EODP
Figure 6-35: Replace the .gitignore file
This will finish the extraction:

99 of 115
Date: 14/09/2020
Subject EODP
Figure 6-36: Sources extraction complete
Press ‘Show the Files’. You can delete the zip file now, or move it somewhere else (you can always
download it again).

100 of 115
Date: 14/09/2020
Subject EODP
Figure 6-37: Once completed, delete the zip file (or move it somewhere else).
6.6.4. Commit the sources

Now that we have the sources in the repository folder, check out the git status. Go to the terminal,
change directory (cd) to the repository folder. Check the contents of the folder (ll) and do a git status.
Notice that all of the files except the README.md are marked in red, and Git tells us at the bottom that
the files are untracked (not present in the repository).

101 of 115
Date: 14/09/2020
Subject EODP
Figure 6-38: Untracked files.
Add all the sources. You do either of these:

git add -A
git add auxiliary common config gm ism l1b l1c LICENSE md sgm
With a git status you get the full list of files added:

102 of 115
Date: 14/09/2020
Subject EODP
Figure 6-39: Sources added.
Now commit & push to origin/develop.
Figure 6-40: Commit.
Figure 6-41: Push to origin/develop.
Head over to GitHub and check that you can see the newly added sources in your repository in the
develop branch:

103 of 115
Date: 14/09/2020
Subject EODP
Figure 6-42: Sources are now in origin/develop.
6.7. Git workflow

You are going to be working alone in your own repository, so the workflow is straightforward. In a
project, there are multiple developers committing to the same repositor, and the workflow is slightly
more elaborate. Git is in my opinion the best version control tool for teams and is the most widely used.
In this section, first, a simplified workflow for a single developer is explained (our case), and secondly,
the classic Git team workflow is presented (for your information only).
6.7.1. Git for a single developer

In our repository we have two branches: develop and master.
❑ The develop branch is the working branch the developer commits to.
❑ The master branch is reserved for milestone deliveries, or certain intermediate stable versions.
Summing up, always commit to develop. From time to time, you can push a stable version to master, for
example after the completion of a module.

104 of 115
Date: 14/09/2020
Subject EODP
Figure 6-43: Git develop and master branches. You will commit to origin/develop. (Optional) you can push
a stable version to master. Source: nvie.com
Moving from develop to master is done directly in GitHub. Go to ‘Pull request’, and click on the green
button ‘New pull request’:
Figure 6-44: Git create pull request
Select base:master and compare:develop. Then click on ‘Create pull request’. You can write a comment
that explains the milestone, or the reason for the pull request.

105 of 115
Date: 14/09/2020
Subject EODP
Figure 6-45: Git create pull request
After that, you should get the option to merge. Click on ‘Merge pull request’:

106 of 115
Date: 14/09/2020
Subject EODP
Figure 6-46: Git merge pull request
And confirm the merge:
Figure 6-47: Git merge pull request
Once merged, you will find the latest version of develop uploaded to master:

107 of 115
Date: 14/09/2020
Subject EODP
Figure 6-48: Git develop successfully merged into master
6.7.2. Git for teams

Where there are several developers, there needs to be a mechanism where each developer implements
“undisturbed” and can safely introduce his/her changes with the following guarantees:
❑ That the latest version of the code is taken into account.
❑ Making sure there are no conflicts.
In the Git team repository, the develop branch is the reference branch of the project. It is the latest
~stable working version of the project. The master branch is usually reserved for milestone deliveries.
To be able to work, each developer has its own branch, different from develop. It is very typical to have
many branches in a git project. Here is an example from FLEX. Notice how I’m developing in the
FLEXFIPSGP-33 branch. Each of these branches are called “feature” branches.

108 of 115
Date: 14/09/2020
Subject EODP
Figure 6-49: Git develop and master branches. You will commit to origin/develop. (Optional) you can push
a stable version to master. Source: nvie.com
A developer works on its own, undisturbed, it their feature branch. He/she will commit to their local
repository and push to the origin/feature branch (upload it to the GitHub feature branch instead of
develop) with (for this example):
git push origin feature/FLEXFIPSGP-33-simplified-scene-generation-implementation
Once a given “feature” is finished, for example a given algorithm has been implemented and unitary
tested, the next step is to merge with develop. This has several steps:
❑ First the developer has to ‘pull’ the origin/develop to its local branch. This means, the latest version
of develop is merged into the local branch. Git does this merge, but it can lead to conflicts that the
developer needs to manually solve.
git pull origin develop

109 of 115
Date: 14/09/2020
Subject EODP
Figure 6-50: Example of “pulling” the origin/develop into the local feature branch.
In the example shown in the figure above, there are luckily no conflicts.
❑ Once the conflicts are solved, the developer needs to commit and push to origin/feature branch.
❑ After this, in GitHub (or wherever the repository is located), the developer performs a pull request.
This means that the developer requests that the changes of the feature branch (which include and are
merged with the latest develop) are introduced in a new version of develop. Another developer or
team member shall review the differences and approve the pull request.

110 of 115
Date: 14/09/2020
Subject EODP
Figure 6-51: A pull request. In this example I’m requesting to merge feature/FLEXFIPSGP-33 branch into
develop.
❑ After the pull request is approved, the developer shall merge the new commits. This way, the develop
is always a clean, stable version. The feature branches are reserved for each developer and they can
have unstable working versions (with frequent intermediate commits so that the work is not lost in
case of a crash).
❑ After a formal delivery of the code, the code is tagged and pushed from develop to master.

111 of 115
Date: 14/09/2020
Subject EODP
Figure 6-52: Git master, develop and “feature” branches. Each developer works in its feature branch. They
will pull the origin/develop branch to their local branch to fetch the latest changes. Afterwards the user
will commit his/her changes to his/her branch, push it to origin/feature branch, and request a pull
request. A different user will approve this request and the changes shall be merged to develop. Source:
researchgate.net
6.8. Git basic commands

There is a lot of documentation online for Git. A good reference page with the basic commands is this
one:
https://rogerdudler.github.io/git-guide/.
Some commands frequently used:

Table 6-1: Some useful Git commands
Command Action
git diff file Shows the differences of a given file with the version in the repository.
git checkout file Reverts back to the version in the repository, discards your local changes.
git add --force files If you want to add a file, for example a text file, that is ignored by the .gitignore,
you can force git to add it.
git add -A Adds all of the tracked changes (all the files marked in red).
git branch To see which branch you are in
git branch -a Additionally, prints all the branches in the repository
6.9. .gitignore file

The .gitignore file tells git what type of files to exclude from version control. For example, typically you
don’t want to include in version control the outputs of your simulation and miscellaneous files like the
contents of the build folder (if you are compiling with C/C++ for example).

112 of 115
Date: 14/09/2020
Subject EODP
In the .gitignore file provided with the sources ([EODP_GUI], section 5.2 “EODP Reference Code”), the files
excluded are NetCDF files (.nc), figures (.png), executable (.exe), logs and text (.log, .txt, .csv), etc.
6.10. Exporting the commit history

Open a command prompt. Go to the project folder and type:
git log 2>&1 | tee log_history.txt
This will output the log history to a text file called log_history.txt
6.11. Exporting the source code

To export your code in GitHub:
❑ Go to the repository page
❑ Got to the develop branch
❑ Make sure you have pushed all of your local changes to origin/develop (see § 6.6.4). Check the
commit history to make sure the latest version is in origin (i.e., GitHub).
❑ Click on the green button ‘Code’ and then on ‘Download ZIP’.
Figure 6-53: Export the source code in GitHub.

113 of 115
Date: 14/09/2020
Subject EODP
6.12. Saving the Git credentials.

To avoid having to retype your credentials over and over again, you can save them. Type:
git config credential.helper store
And then perform any git operation that requires authentication, for example a ‘push origin develop’.
Check out the .git-credentials file in home (/home/luss/.git-credentials), where you will find your
credentials stored.

114 of 115
Date: 14/09/2020
Subject EODP
END OF DOCUMENT

115 of 115

EODP Environment R1.0

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

EODP Environment R1.0

Uploaded by

Copyright:

Available Formats

Environment

Earth Observation Data Processing

Name Function Signature

Approved by Lucia Soto Coordinator of EODP lsoto@ing.uc3m.es

MASTER IN SPACE ENGINEERING

1.0 - First issue 14/09/2020

MASTER IN SPACE ENGINEERING

MASTER IN SPACE ENGINEERING

4.2.3.1. Create a file (nano) _____________________________________________________ 64

MASTER IN SPACE ENGINEERING

6.8. Git basic commands ___________________________________________________ 112

MASTER IN SPACE ENGINEERING

Figure 3-22: VM settings. Shared Folders. ..................................................................................... 28

MASTER IN SPACE ENGINEERING

Figure 3-61: Notepadqq .............................................................................................................. 49

MASTER IN SPACE ENGINEERING

Figure 4-22: Example of locate ..................................................................................................... 70

MASTER IN SPACE ENGINEERING

Figure 6-30: Unzip ...................................................................................................................... 94

MASTER IN SPACE ENGINEERING

MASTER IN SPACE ENGINEERING

2.1. Applicable Documents

Ref. Code Title Issue

2.2. Reference Documents

Ref. Code Title Issue

ATBD Algorithm Theoretical Baseline Document

COTS Commercial Off The Shelf

EODP Earth Observation Data Processing

FYI For Your Information

ISM Instrument Module

L1B Level-1B Processing Module

L1C Level-1C Processing Module

MASTER IN SPACE ENGINEERING

3. SETTING UP THE WORK ENVIRONMENT.

3.1. Install Oracle VM Virtualbox

Figure 3-1: Download page for VM Virtualbox

Click and save the executable somewhere:

MASTER IN SPACE ENGINEERING

Figure 3-2: Save executable for VM Virtualbox

Figure 3-3: Installation of VM Virtualbox

MASTER IN SPACE ENGINEERING

Figure 3-4: Installation of VM Virtualbox

Figure 3-5: Installation of VM Virtualbox

MASTER IN SPACE ENGINEERING

Figure 3-6: Installation of VM Virtualbox

Figure 3-7: Installation of VM Virtualbox

MASTER IN SPACE ENGINEERING

Figure 3-8: Installation of VM Virtualbox

Figure 3-9: Installation of VM Virtualbox

Finally, open Virtualbox (double-click on the newly created Desktop icon):

MASTER IN SPACE ENGINEERING

Figure 3-10: VM Virtualbox

3.2. Import the Virtual Machine

MASTER IN SPACE ENGINEERING

Figure 3-11: VM Virtualbox. Import Appliance.

MASTER IN SPACE ENGINEERING

Figure 3-12: VM Virtualbox. Import Appliance.

MASTER IN SPACE ENGINEERING

Figure 3-13: VM Virtualbox. Import Appliance.

MASTER IN SPACE ENGINEERING

Figure 3-14: VM Virtualbox. Import Appliance.

Modify, if you want, the destination folder for your VM: