Fe 10 or Youd QD : we °, _ Roo! Cd uae nS tcpdump! ! C = N . OY +/ oO wizard * A SMALL = 001: HANDBOOK FOR INYON@ WHO WRITES (OR RUNSII) PROGRAMS ON Linux COMPUTERS BY: JULIA EVANSHi! This is me: C) Sucia Evans blog: jwns.ca twitter:@ bOrk and in this 2me T want to tell you about how Z got better at debugging These are 5 ways I've changed how Z tink about debugging: Remember the bug, is happening for a Vogt cal feason. Theres no magic. 4 Be confident can fix it ° maybe this before. is teo hacd} now: G Talk to someone [OS ° ~~Q > =pd name: “Frutu , Ge 1 " program {rere “bloe " cat d= 94 request Every programming language uses the Same net work protocols | So the network is a mee language - independent place to answer questions like ° ° was the cequest wrong, or was it the response’ 2 “Us muy server even ons * my program 's slow, Whose Fault is it? Let ‘ go Yvee \ ve a r= wt xe “x netcat yr %, Oy ors 1 0, vr HTTP request S are Fundamental! really simple — they're yest text! To seb - that, let's make one by hand First, make @ 7 TT Giles | recuest txt | a tote te ' Get / H1TP/1.1 . Host: ask.metafilter.com { User-Agent: zine Wy \ (2 new lines! important!) | Then? ne gtands For net cat '¢§ cat request. txt | no metafilter. com 8O |} You should get a response back with a bench 3 fe of HTML! You can also use netcat to sen huge Si les over a local netuowk quickly : step! + fon target machine) tep 2: (on the source ) r ! $ hostneme - = ' 192.168.2132 --- L$ne -4 94317 Sigh a - - = dhis listers on the port - this sends the data 5 \ ne 142 168. 2.182 1131!& netstat & Every network request gets sent to a port (like 80) on a computer. To recewe a request, a program (aka"s erver’) needs to be listening on the port. Finding Gut which programs are Listening on hich ports is really easy . t's just xo R “4 \ v " vote una, please § & Met _— also known as ye ‘ Sudo netstat -tonap ( \ —-S- ee Kee Here!s what you'll see: prote local address PLD / program name tcp 0.0,0.0 8353-2493 / python port 2 So! LT @& netstat because it tells me which processes which ports. instead - are conning On ei On 05K, use 'Isof -i-P}nare narep is my Favourite starter network Spy tool | Try it right now! Run? sudo ngrep -d any metaFilter Then go to http:/ metafilter. com in your browser. You should see matching network packets in ngreps out pu T We are SPIES NU Recently at work I'd made achange te a clrent so that it sent {“special - td: 4 with all ts requests. T wanted to make sure Twas working, so L ran eee ee Oreos ~~ ee ESE eS I found out that everything Was ok RU) 3 S% + codump ¥Y tcodume is the most difficolt networking tool we'll discuss here, and it took me a while to Y it. ZT use it 40 save network traffic to analyze latec! Awesome thing “port 8999" is actually a ting, program in the Sudo tepdumo pork 8947 \ -w . \V| Berkeley Packet _*‘ Ssevree cap 4 Filter” (Pr?) Zz language. BPF 7 Filters get compiled and they con ceally a” pcap File” ("packet capture") is the fat Standacd for saving network traffic. Every Abing vadeeshand pcap@ Some situations where I'll use tcpdump 2 - Lin send Ing a request to a machine and L want to know whether ij!s even geHing there (Sepdome port 80} will print every packet onport 60) —T hove some slow network comnectiong and Z want to know whether to blame the cheat or servec. (we'll also need wireshack!) - 1 jest want +o print out packets to see them (tepdump -A)wire shack Wweshark is an GUL tool Sor network analysis, Here's an exercise to Yeacn it¥ Run this: ‘ sudo tepdump port 80 -w hHp- pcap ‘ While that's Funning, open metafiller.com in your browser. lov jvns.ca | - Then press Ctrl+C to stop tepdump. Now we have o peap! | wweshack http.pcap ! Explore the Wireshark interface! Questions you Can tru to Qnswer: OQ What HTTP headers did your brouser Send to metafilter. com? Chint: search Frame contains “GET"!) ® How long did the longest request take? (hint: click Statistics -> Conversations) @ Hou mony packets were exchanged . 1 P addtess4eom with meta Filter.coms server? [Leng meatier canes Saueeas) (hint: search ' cp.dst == SH66 OEsection 3: CPU Ft Your programs spend a lot of time on the CPU! Billions of cycles. What are they DOING?! This section is about using oer $ to answer thet question, a Linvy-onl tool that is actremelu useful and not 45 well-known as it should be vs (in general, my aim inthis zine is to showcase tools that L don't think get enough YOY) Some things I didn’t hove space Sor in ths section bet I want to mention any war - valgrind “the Save ecosystems fantastic fools (jstack, VisvalM, Mission Control, Your kit) which your language is probably jeclous ot Fterace (or Linux kemel perf problems) -eBPFY) pert VY pect is not simple or elegant. It isa werd mult itool that cloes a bunch of different, very vsesul things. Furst! 4's ar sampling " profiler /- Tey funning : ao TPs - { Saves q (press Ctel4C after 2 seands) «pe it " Uy seeds, every few milhseconds : y e python process is doin . Lel's See the results | J jee i$ Sudo perf report , Mine Says it spent 5%, of ids time in the PyDict- GetItem function. Cool ! We learned a tay thing about the CPython interpreter | Works everywhere Y pecf can be installed on pretty much any, Linux machine. The exact Features thas will depend alittle on your keenel version. af you're a Python /Roby/ Sava/ Node programer, you might be getting antsy. “ZI want to know which Ruby Function is running | Not the C function! Stick uth me though. I get you,pect is For everyone One day, L had a server that was using 100% of its CPU. Within about 66 seconds, T knew it was doin regular expression mat chi in Ruby. Hou? oo “4 i § sudo pert too \L \ PrecesS PID Yo Fonchion ruby \9SF FSH match -at l —-—---3 ‘ 7 2, pect top doesn't always help- & Ruby's internal & Fac Fromit. But id's an eas regeep matching Func tool to tr4, and 4 SEP matches Fondian 's awesome when it does help. DIL vee especially Sava and node devs ¥ Remember when L said pert only knows C functions? Lt!s not quite true, node.js and the SVM (java, scala, clare...) have both taught perf about their functions . ~ Use the Look vp * perf -map-agemt’ -- perf basic -prot on GitHub and Follow command Ine option the directionsspy, on your cPuY Your CPU has a small n tip cache onit (the ua cache ) Joog that it can access | n VO.S Nenoseconds ! faster than RAM le “Latency, Numbers every Programm er » should know : LF you'e trying to do an operation 1m MICO Second$, CPU cache usage maHers! how do I know iF my program is Using those caches? /\ Nou to «| pect stat -e L4-dcache-loacl-misses } use it | hs 1 This cons ‘Rs’ ond prints a report at the enc. y how it Your CP can keep all kinds oF countess works | obeut what th doing. spect stati asks it to couat things 4 then collects the tesult. _ Harduare is cool . Krew more about how Your harduare works can really par off & SSSF lame graphs Bee Flamegraphs are an awesome wa to visualze CPU performance , popularized bi Brendan Greggs Flamegraph. p! tool. Here's what they lock like: fl " U4 T) [ ) alligator 86% main 100 7» $$ ain 100 te Theyre constructed From collections lusvall $ thousands) of stack traces sampled From 4 program, Th is one above means 80%, of the stack traces started with “ allisgator and 10% with ~ ‘Dende ‘ _ ect You can constevet them from perf recording s (look vp” Brendan Gregg Flamegraph” for how) , but lots of other unrelated tools can produce them too. I © them,T hope you leacned Something new, Thanks for reading 9 Thanks to my, partner Kamal for endless Ceviews , to the Gmaziney Monica Dineulesco (@ notwaldort) For the cover art, and many others. Tf you want to know more — ru, site has a lot (jvns.ca) and brendan gregg com does too. But really you just need to experiment. Try these tools everyshere- See where they help and where they don't. It takes a lot of prachee to use these tools to debug real problems, L've been learning them for 2 years, and T've gotten pretty Far, but there's alone Way to ge. It's ceally Fon UUSLiked this? You can print more \ foc Free QD = http: //jvas.ca /zines = CC- BY-NC-SAN-O wizard debugging industries 9