CCPR Computing Services

Workshop 1:
Programming Basics, Unix, Remote Computing
October 13, 2004

Part 1: Programming Basics

Motivation
Before you start coding
Programming Conventions

Documentation
Names, comments

Directory Structure
Basic Constructs
Miscellaneous (debugging, cross-checking
results)

Motivation

Facilitate research
Save time
Cleaner code
Easily share programs
Basic Concepts
MUCH better programming

Programming Conventions

What are conventions?

Who cares?

Examples

Readability of code
Organization
Transferring code to others

Apply conventions consistently to:

variable names and function names

comments
directory structure

Before you start coding

THINK
WRITE down the problem
WRITE down the algorithm in English (not code)

Modularity
Comments
Create test (if reasonable)

TRANSLATE one section to code

TEST the SECTION thoroughly
Translate/Test next section, etc.

Documentation - File Header

File header includes:

Name
Project
Project location
Date
Version of software
Purpose
Inputs
Outputs
Special Instructions

#Laura Piersol (laurapiersol@ccpr.ucla.edu)

#HRS project
#/u/socio/laurapiersol/HRS/
#October 11, 2004
#Python version 2.4
#Stata version 8
#Purpose: Create and merge two datasets in Stata,
# then convert data to SAS
#Input programs:
# HRS/staprog/H2002.do,
# HRS/staprog/x2002.do,
# HRS/staprog/mergeFiles.do
#Output:
# HRS/stalog/H2002.log,
# HRS/stalog/x2002.log,
# HRS/stalog/mergeFiles.log
# HRS/stadata/Hx2002.dta
# HRS/sasdata/Hx2002.sas
#Special instructions: Check log files for errors
# check for duplicates upon new data release

Naming Conventions

Not a detail!
Good names clarify your code
Portray meaning/purpose
Adopt a convention and BE CONSISTENT

Naming Conventions, cont.

Use language standard (if it exists)

If no standard, pick one and BE CONSISTENT
Functions:
Scalar variables:
String variables:
Global variables:

getStats, calcBetas, showResults

scPi, scGravity, scWorkHours
stName, stCareer
_Country, _Nbhd

Be aware of language-specific rules

Max length, underscore, case, reserved words

Naming Conventions, cont.

Differentiating log files:

LogWt vs. var1

AgeLt30 vs. x

Procedure that cleans missing values of Age:

MergeHH.sas, MergeHH.do
MergeHHsas.log, MergeHHsta.log

Meaningful variable names:

Programs
Log files

fixMissingAge

Matrix multiplication X transpose times X

matXX

Commenting Code
Good code is SELF-COMMENTING

Naming conventions, structure, header explain 95%

Comments explain

PURPOSE, not every detail

TRICKS
(good) reasons for unusual coding

Comments DO NOT

fix sloppy code

translate syntax

Commenting Code - Stata example

SAMPLE 1
program def function1
foreach v of varlist _all {
local x = lower("`v'")
if `"`v'"' != `"`x'"' {
rename `v' `=lower("`v'")'
}
}
end
SAMPLE 2
*Convert names in dataset to lowercase.
program def lowerVarNames
foreach v of varlist _all {
local LowName = lower("`v'")
*If variable is already lowercase,
*rename statement throws error.
if `"`v'"' != `"`LowName'"' {
rename `v' `=lower("`v'")'
}
}
end

No conventions, comments, structure

Comments: succinct and not overdone

Names:
lowerVarNames
-action word for program
-distinct use of case
LowName
-descriptive
-distinct use of case
v
-looping variable and short scope
-non-descriptive, but does not detract
from meaning!
Structure- indentations, parentheses lined up!

Directory Structure

A project consists of
many different types of
files
Use folders to
SEPARATE files in a
logical way
Be consistent across
projects if possible
ATTIC folder for older
versions

HOME
PROJECT NAME
DATA
RESULTS
LOG
PROGRAMS
ATTIC

Miscellaneous Tips

BACKUP! Weekly zip file stored externally

README.txt file to describe folder
BE ORGANIZED
CROSS-VERIFY results
Something not working?

Remember the computer is following YOUR

directions go back to your code

Programming Constructs

Tools to simplify and clarify your coding

Available in virtually all languages

Constructs - Looping

Repeat section of code

START value, INCREMENT, STOP value
Example

convert uppercase to lowercase for each variable

in a dataset

Constructs Looping Examples

C for loop: Start with x=1, Increment = x+1, Stop when x==10
for(x=1; x<10; x++) {
code
}
PERL while loop: Start with count= 1, Increment= count+1, Stop when count==11
$count=1;
while ($count<11) {
print "$count\n";
$count++;
}
STATA foreach loop: Start = first variable in varlist, Increment = next variable in
varlist, Stop =last variable in varlist
foreach v of varlist _all {
local LowName = lower("`v'")
if `"`v'"' != `"`LowName'"' {
rename `v' `=lower("`v'")'
}
}

Constructs - If/then/else

Execute section of code if condition is true:

if condition then
{execute this code if condition true}
end

Execute one of two sections of code:

if condition then
{execute this code if condition true}
else
{execute this code if condition false}
end

Constructs - Elseif/case

Elseif - Execute one of many sections of code:

if condition1 then
{execute this code if condition1 true}
elseif condition2 then
{execute this code if condition2 true}
else
{execute this code if condition1, condition2, condition3 are all false}
end

Case- same idea, different name

case condition1 then
{execute this code if condition1 true}
case condition2 then
{execute this code if condition2 true}
etc.

Constructs - And, or, xor

AND OR XOR -

BOTH conditions must be true results in True

AT LEAST ONE condition must be true results in True
EXACTLY ONE condition must be true for statement to be true
1

AND

True

AND

False

AND

False

OR

True

OR

True

OR

False

XOR

False

XOR

True

XOR

False

Constructs - Break

Stop execution of program

Examples:

Debugging. If particular error occurs then break.

Parameters in function call are nonsensical. Print
error and break.

Constructs - keywords

Looping - for, foreach, do, while

If statements if, then, else, case
And/Or/Xor logical, and, or, xor, &, |
Break exit, break

PART 2: Unix

Motivation
Basic Commands
Job submission and management
Pipes
Unix Shell
Script files

Unix

Motivation
A quick history
Unix variants

(AIX, Solaris, FreeBSD, Linux)

Where?

Nicco (SSCs server)

CCPRs linux cluster (coming soon)
CCPRs data server (coming soon)

Unix Basic Commands

Getting Help
man command

list help for command (man if short for

manual)

man -k command

keyword search for command

whatis command

give a brief description of command

apropos keyword

list commands with keyword in the NAME

section their man page

Unix Basic Commands

File Management
ls

list files (options l for long, -a for all)

mv filename1 filename2

rename filename1 with the name filename2

cp filename1 filename2

make a copy of filename1 and call it filename2

rm filename

delete filename

more filename

print contents of filename1 to the screen

cat filename

print contents of filename to the screen

cat filename1 >>

filename2

append contents of filename1 to the file filename2

cat part1 part2 >>

bothparts

appends the contents of file part1 and file part to to the

file bothparts

head filename

show first 10 lines to screen

tail filename

show last 10 lines to screen

Unix Basic Commands

Directory Management
pwd

show current directory

mkdir dirname

create new directory dirname

rmdir dirname

remove directory dirname

cd dirname

Change to directory dirname

cd

change to your home directory

cd ..

move one directory up

cd ../..

move two directories up

home directory

cd ~/scripts

change to the scripts folder in your home directory

Unix Basic Commands

Using Previous Commands
!!

repeat last command

!v

repeat the last command that started with v

!-2

repeat second to last command

arrow up/down

scroll through list of previous commands

history

list history of commands used

Unix Basic Commands

Other Useful Unix Tools
*

wildcard (matches any number of characters)

wildcard (matches single character)

grep word filename

list lines of filename containing word

diff filename1
filename2

shows differences between filename1 and filename2

wc filename

counts number of lines, words, and characters in

filename

sort < filename

sorts the lines of filename

who

lists users currently on the system

cal

displays current month's calendar

date

displays date

Unix Basic Commands

Disk Usage
du s

Total kilobytes used in current directory

du a

Same as above, but more detail

ls l

Gives individual file sizes in bytes

Editing Files in Unix

Vi Editor and Emacs

Neither are user-friendly for starters

Look at CCPR internet when you start
Best way to learn is to start editing a document
Once you get used to them, theyre easy and fast to
use.

Being nice

Always run your jobs nicely

Prevents interfering with other users
Precede command with nice +19 (no quotes)
user@nicco%nice +19 stata b jobfile.do

Job Submission in Unix

Interactive
user@nicco%stata

Foreground jobs
user@nicco%nice +19 stata b do jobfile.do
user@nicco%nice +19 sas jobfile.sas

Background jobs &

user@nicco%nice +19 stata b do jobfile.do &

user@nicco%nice +19 sas jobfile.sas &

Background jobs with logoff nohup

user@nicco%nohup nice +19 stata b do jobfile.do &
user@nicco%nohup nice +19 sas jobfile.sas &

Job Management
Ctrl-c

Cancel a foreground job

Ctrl-z

Suspend a job in the foreground

bg

Move a suspended foreground job to the background

ps u

List information for processes you own (under current shell)

ps ux

Lists information for process owned by you and others

ps aux

Lists information for all processes (including root, bin, etc.)

ps aux | more

Display output of ps aux one page at a time

ps aux | grep PID

List lines from command ps aux containing PID

kill PID

Kills (cancels) process number PID

top

Lists 15 processes using the most cpu processing power

Stops command top

Pipes

| redirects
command 1 output

command 2 input

Command

Output from Input to

Result

top | grep piersol

top

grep piersol

Jobs in top 15 containing piersol

ps -aux | more

Ps aux

More

Jobs listed one page at a time

ls | wc -l

ls

Wc l

Count number of files and

directories in current directory

Extends to more than 2 commands

Unix Shell

Whats a Unix Shell?

Whats a shell script?

The Unix shell is the program that provides the

interface between the user and the kernel
A list of commands put into a file that can be
interpreted by the Unix Shell.

What are scripting languages?

Generally easier to code but less efficient

Shell scripts, Perl, Python

Remote Computing
Windows to Whitney

Remote Login

Remote Desktop
Connection
>= XP included*
< XP download*

Web-based
Remote Desktop
Connection*

File Transfer

Map a Drive via

Windows Explorer

*See http://www.ccpr.ucla.edu/asp/compserv.asp, XP users get latest version

Remote Computing
Windows to Unix

Remote Login

SSH Secure
Shell Client*

File transfer

SSH Secure
File Transfer Client*

* http://computing.sscnet.ucla.edu/training/tutorial_SSH.htm
** http://computing.sscnet.ucla.edu/training/tutorial_samba.htm

Map a Drive via

Windows Explorer
Need Samba account**

Finally

Questions and Feedback