sar0te COC print (n); nttj} while (n<(n+1)){£ Home 810 Troubleshooting Common Networking Problems with Wireshark, Pt. 6: Finding the Source of Network Delays [Author's Note: Ths isthe lat par in a separ series about finding ane solving many networking anomalies “sing the Wiesback metwore protocol analyzer. you are ew tothe sere, you ca fing part | hace and he tahoe series here, Irmay seem surprising inthis age af ulqutous GignbtChernet connections, but many af he cll in Enterprise networking stl revolve around network delay problems, Slow access fo websites, slow logon, slow ‘mall all ofthese sues can potently be related to retworkgelys, Unvorunatey, tracking cow the source of metwor delays can be very Fustrating at times. With the ght ikerng, Mowevr, you can eliminate much of ‘he rustraton an dll dwn tothe souree. In Sore cases, the delays wil be easy Lo Spot, showing up asa per-packe delay in the traces In others, the packet tency maybe very small Bu the upper lve aency may be huge. The simplest eck, Maeve, 'S typically per pace: Iateey. Ths canbe done eas in Wireshark by fitering downto communications ‘beeen te wo problematic systems. Once th done, ensure that your time sialay formats seo "Seconds since previous packet inthe view menu, an elk onthe Time column heading to sor by time. ‘This wll show you you longest delays between packets you have many packets with vey long delays (02 seconds or niger re-sort by frame nimler fr several of tree ane examine the wath The typeof trafic Ul determine the delay is a problem. For instance, along delay on TTP keep-ave i nota proslen, but {ong delay Deoseen an HTTP Get and HTT? 200 response Note: Wake sure, when ooking fr delay issues, that the wace was captured with a minimal amount of eaptire skerng. The deft etng of fiterng out RD? i fie (unless you are {his stat fyou are fering outs rg chun ofthe vate, yu ae in eect cresting $395 However, once you find the delay you may sil aed to ig further to locate the source, For example. you right click on an HTTP packet in Wireshark and choose the onion Fallow TCP strearr’ Wreshark wil fier based on tye source and destination sockets. giving you's good summary ofthe TC> conversation. AN ‘example of 2 ceived conversation Is shown below No. [Time | Source Destination V1 0.008281 160.207.226.48 160.207.1241 HTTP 12. 0090771 160.207.1241 160,207.226.88 HITP 113, 0000883 160.207.1241 160.207.226.88 HTTP Protocol Info (ET fdefasteaspx HTTP). HITP/1s1 AD) Unauthorized text ntmb Continuation or nor-HTTP trafic 2456 > hp [ACKT Sea=s91 Ack=1677 Wie=85520 Leno GET Jdetat aspx HTTP) NTLMSSP_NEGOTIATE HITP/1.1 40) Unauthorized, NTLMSSP_CHALLENGE (exter Cominuation or nor-HTTP ttle 2456 > hi ACK] Ses=1283 Ack=3651 \win-85520 Leno GET deft. aspx HTTP) NTLMSSP_AUTH, Use: DAIS Michael vost ep > 2456 [ACK] Sea=B651 Ack=2143 Win=s4675 Lenao HTTP/T-1 200 Ox Gextihe 114 0.000986 160.207.226.485 160.207.1281 TCP 115 0997079 160.207.2648 160.207.1241 HTTP 116 a.991008.160.207.12.41 117 0000562 160,207, 160.207.726.48 HTTP 118 0.000993 160.207.2268 160.207.1241 TCP 199. 0008675 16020722648 160.207.1241 HTTP 121 0.112828 160.207.12.41 160.207.226.481? “Troublesnoting Common Networking Probl with Witeshark, Pt. 6: Fining the Source of Network Delays « Courting toy ————_ ABOUT ME Brian Hill POPULAR POSTS robles with Wireshark, Pe? TCP Mansel conaeat Netancin eet ae Souice of Network Delays RECENT POSTS. est Housenarring Git EVER In Praise of he Roguelee Fool vs ATRI=8 Cross-Country ances ‘cms Seon] RECENT COMMENTS It on Frogram Review.toss barb ‘Nex on Program Revien-Boss Barbell DOmitry on Pagram Revew-Boss Rael} Srenges Method (on 3c Dmiry on rasram Aeaswtors abl in Method: ‘Suenass Method (an act ‘any broaress 5x3 G1500 ICNP. Series Troubleshooting Common king Problems with Wireshark Wireshark articles android metal future nexus gn prediction ‘eligion ars technica dual booting ‘ues Apocalvnse video game strana reaching cisco travel heath belies bq hhetwark design education Linux Gue Postecineal [shine x Dan Brow hpscounting--irvinty.com/201206Irovedeshooing-commer-networking prabloms-with-wieshark-pl-6-inding-the source-a-ratwork delays! wesar0te “Troublesnoting Common Networking Probl with Wireshark, Pt. 6: Fining the Source of Network Delays « Courting toy ooking at his conversion, you may correctly surmise that the page in guestion takes ong toe ta revive (Ground 1.7 seconds) but we sll co rottrly Know the source. Even worse, since an HTTP page can include ‘ull references to ad onal les (Such as mages), which can extend over Multiple TCP sessions ooking ta singe steam does not necessarily gve us a ue appre ation ofthe delay. Te see the ul delay, we ust ‘ter us on the two addresses inthe conversation along withthe prtecal TT, ke so. (paddr==160.207 226.48 and ipadds==160.207 12.41) and hip With thi ter, we can fok for eacn HTTP Get request and respon, and determine wen the ls reauest for ‘he page is set snd when the response for that requests returned. We fir thatthe las request or the fe erossbulet gn rame 3962, hich Is retuned in fste 4285, We know th is the let request because ‘the Get request for default aspx (and all zh Get requests elated to ths page) repeats indicating a new page Fegues afer this Ge request Nw that we Know where the conwesation ends, We cn ft! tus hs HETP Conversation by using the flloning titer. (prada eg 160.207 22648 andipadcr eg 160207.12.41) and hip anéframe.number <= 4285 With this iter applied, using ne Surmmary command under the statisti ment gives us the following ‘nformtion © Ethereal: Summary > 2965709 bytes boca (tcpduro, Ethers), ec.) Packet zeit 53S bytes Tae rst packt: 2005-1228 14:45:08, Last packet: 208-1228 14:46:47 aps const Display Diplay ter: (poder eq 160.207.206.49andip.ad oa 160,207, 12.41) anda and frame.number <= 4205 Marked packets: 0 Tete Centred Dla Between fst andes packet 101,087 sec 28.524 see Packets ear eas ‘avg pachatsfsec erat 24.88 vg chat se ‘20.000 bytes 1171,680 bytes Bytes zerseis 607300 va, byteseae zaueaqee — 28002.086 vg, MB sec ozs 0226 [As te summary shows, tere 326.5 second dela In retteving the page. This delays huge, much worse ‘han we previously arepsted 20 now we need to ain urtherto determine why. Another Wireshat fier can help neve {prada eg 160,207:22648 and ip.ader eg 160,207.12.41) ane (htp.request or mtpcesponse) ard framesnomber =~ 4245 ‘Ths ere essencallyAkering tal rac betwen the sw host tha | also an HTTP request responce packet and hs a frame number less than 4285, Tvs ites the HTT? continuation packets ut ofthe ace, ‘rapping the trace rom B29 frames to 182 frames, However, we can break t down further and ook for just ‘heh gh delay packets oy adding ‘rame.cere_ detache ike, ike 50 Up.adar eg 160,207.22648 and ip.ader eg 160,207.12.) ane hep reques or Mtpwresponse) ane irame.nomber <= 4285 ane frametimecdelt »= 02 This erie the same a5 te previous “er, but row we are adding the cordon that al faves displayed must have atime deta from the previously csplayed frame equal fo or greater than 200m, Ths fikes the ‘Fame dows 1027 high-lateny ames: No, [Time [Source Destination | Protocol Ine 1) === 16020722648 160.207.1241 HTTP GET /defauaspe HTTPITA 165 1.765202 160.207.1241 160.207.226.68 HTTP —_MTTP/1-1 200 0K (ext/miml (GET /layous/1033owbrows js 244 0.208087 160.207.2648 160.207.1261 HTTP —HITPIDAI 351 1.450849 160.207.12.41 160,207.226.48 HTTP HTTF/1.1 200 OK ext/cs) hipscounting-t-iinty.com201206Irovedeshooing-commer-networking prabloms-with-wieshark-p-S-inding-the source-o-ratwork delays!sia2016 “Troubleshooting Common Networking Probl with Witeshark, Pt. 6: Fining the Source of Network Delays « Courting toy 550 1.169115 160.207.1241 |160,207.226.68 HTTP HITP/1<1 200 OK (exticss) ATTP/1.1 200 0K applicat one TTP/1-1 200 OK applica one 72k 1.081409 160.207.1261 160,207.226.68 HTTP javascript 920 0.266885 160,207.226.48 160.207.1241 HIT? GET lnyouts/1048/seaenys HITPVI.) ATTP/1.1 200 0K applica on/x= 1316 1.771438 160.207.1241 160.207.226.48 HTTP HTTP/I.1 200 OK textes) layouts images icongod2 st 1282 0.206753. 160,207.226.48 160.207.1261 HTTP TTPAIA 1576. 2.916854 160.207.12.41 160,207.226.68 HTTP HTTP/1.1 200 OK (CIB) AATTP/1.1 200 0K appicatonix= 5 | ayouts images GOAISSoure git AATIP/\1, NTLMSS?_AUTH, User 2002 1.160457 160.207.1241 160,207.226.48 HTTP HTTP/1.1 200 OK Wext/css) 2052 0.200265 160.207.1241 160.207.2268 HTTP NTLMSSP.CHALLENGE (ext ht 2158 1.139176 160.207.1261 160.207-226-¢8 HTTP HTTP/I-1 200 OK (CII) 2226 1.108149 160207.12.41 160,207.226.48 HTTP HTTP/I.1 404 Not Modtfieg 2347 1.28181) 160.207.1241 160,207.226.68 HTTP HTTP/|.1 200 OK (ext/x-componens) 2469. 1.267644 160.207.1241 160,207.226.68 HTTP HTTP/I.1 200 OK (CFB) 2512 1.26884) 160.207.1261 160,207.226,68 HTTP HTTP 161 200 OK (CFE) 2551 0.217619 160.207.12.41 160.207.2268 HTTP _HTTP/I.1 200 0K (F893) 2615. 1.331625 160.207.12.41 160,207.226.68 HTTP HTTP/1.1 200 OK (CIF89S) 2693 1.114701 160.207.1261 160,207.226.68 HTTP HTTP, 1e1 200 OK (CFB) 3290 1.147739 160.207.1261 160,207.226,68 HTTP HTTP 1 200 OK (CFR) (GET layouts iimages|parcradp.t 44282 0.91976) 160.207.12.41 160,207.226.48 HTTP TTP/I.1 404 Not Found (ext/ht) {this pont, we simply nee to [ook for a patter, and lucky the te shows us the pater very cull ‘appeats that realy all of Ure HTTP responses ave Glas of | second or longer. This sa huge nd. ane Domts us very dose rote cause Intl, we maybe tempted to pont to the sever elf s the cause ofthe delay, but we can’ prove that thou ooking atthe serve aide trace. So we wil open the server ade trace and attempt to ind «| necessary he problem Is caused by a serve sie sue, any conversation that going to the same URL on {he same servers approximately the same tne should show simlr dely, and for tne same reasons. To find a sar delay we wil simply mot the oresous Fite sight, removing the Fame number restrtin, Ike #0 (pada eg 160.207.2648 and ip.ade eg 160.207.12.41) ane (ho request o Mtprespense) and ftamestne.dska > 02 “Thi fier vl show us some frames wih vet sing smilatiyto the ones Seen the previous trace: No Time Source Destination Protocol | nfo GET |nyours 1088 owsdrows js HITP/1.1 200 OK application 3751 1.095843. 160.207.12.41 160.207.226.468 HTT? —_jvasernd TTP; 200 OK appiation x= 3856 1081487 160.207.1241 160.207:226-48 HTTP jvascring HTTP/1-1 200 OK appiation/x= ‘This proves that he problem isnot due to ceiy in the link, but s someting related othe web server's alvery of he requerted page. To dg deeper, we now need to select an example othe problem. sy he Get request fr onsbyowss andthe response Sees in fares 8278 and 8751. resaecively ana ltr tall vate Detween this request and response. Ths ste see any adonal wae sat our previous ters may have idea. To dots, we wil use the oting fier “This iter fiters don tol frames between thes two numbers, including the fst and la frames (request Tight wh ooking 3 the frst 36 packet, however, we can Begin fo determine what curing apt ning o-ininty.com/20'20BIrouesheoing-commen-networkingprcblems-with-wireshark-p-6-nting-the-source-d-network delays! a6rote pn 7s 376 na a0 na ne sess mse we v0 p31 292 a p98 295 296 na 96 a9 3300 6.466432 0.000017 0.000018 0.000348 001s 0.000026 0.000012 0.000019 0.000721 0.000826 000439 0.000015 0.002 0.000152 o.000854 0.000017 0.000016 0.000015 o.001842 0.000052 0.000805 0.000022 0.000009 “Troubleshooting Common Networking Problems with Witeshark, Pt. 6: Fining the Source of Network Delays « Courting toy 160,207.226.48 160.207.1287 160.207.1218 160.207.2268 160.207.1219 160.207.1237 160.207.1218 160.20712.19 160.207.1287 160.207.12.19 160,207.12.19 160.207.226.48 160.207.12.41 160.207.1237 160.207.1237 160.207.12.19 160.207.1219 160.207.226.48 160,207.12,37 160.207.1237 160.207.12.37 weo.20712.41 160.207.12.19 160.207.1237 160.207.1241 160.207.1287 160.207.12.19 160.207.12.37 160.207.12.37 160.207.1219 160.207.1237 160,207,12.37 160.207.12.41 160,207.226.48 150.207.226.48 160.207.1219 160.207.12.19 160.207.12.37 160.207.1237 160.207.12.41 150,207,12.19 160.207.12.19 160.207.12.19 weve rer rer rer rer rer rer rer 1. rer avr hme rer re 1 rer 1. rer rer nfo vin "as > a7 ee enon "a9 78 5 Ac e002 Nace) we 007 ease 10762 192 Ae Sa 27082 Chk NCORET] Led T1440} aoe oxi) 4076 > 1292 (PSH, ACK] Seq~27068 ck 602450 Win-65535 TC? ‘CHECKSUM INCORRECT] Len=122 1292 > 4076 [ACK Seq-602450 1292 > 4076 (SH, ACK] Seq_ 603310 Ake 27185 Winn 64925 Len=298 4076 > 1292 (ACK Sea=27185 Acka604208 Winn65535 TC? ‘CHECKSUM INCORRECT) Len-0 Wine has 160.207.12.1847 Tell {4075 > 1292 (PSH, ACK] Seq=27185 ‘Ack 604208 Win-65535 [TCP ‘CHECKSUM INCORRECT] Lene22 1299 » 4076 (ACK) Seq-£04208 eke 27207 Wine 64803 Len= 1460 1202 > 4076 (SH, ACK Seq-608668 4076 > 1292 (Acx) $e9-27307 ck 605966 Win=65535 ITC? ‘CHECKSUM INCORRECT] Len-0 {4075 > 1292 (PSH, ACK] Seq 27307 ‘Ack= 608066 Win=65838 [1CP ‘CHECKSUM INCORRECT] Len= 122 GET | tyouts|1035/owsbrows js HTTP/I-1, NTLMSSP-NEGOTIATE HITP/I.1 401 Unauthonzed, NTUMSS®.CHALLENGE (ext hem Contnustion ar non-HTTP traf 1292 > 4076 [ACK Seq605966 ‘ek=27429 Win 64681 Lan= 1460 1292 > 4076 [SH, ACK Seq=607426 eka 27429 Winn 6681 Len=295 4076 > 1292 (acx| Seq-27429 ‘ek 607724 Win-5535 ITCP ‘CHECKSUM INCORRECT] Len=0 {4075 > 1292 (FSH, ACK] Seq 27409 ‘ek 607726 Win-65535 [1CP ‘CHECKSUM INCORRECT] Len= 122 2466 > hp LACK) Sea 1585 ‘ek 2651 Win 65520 Le 1292 > 4078 [ACK| Seq607724 eke 2755) Win 64559 Len-1450 1292 > 4076 [ACK] Seq609184 ‘ek 2755) Win 64559 Len-1460 1202 > 4076 [ACK Seq=610548 ka 2755) Winn 64589 Len=1460 4076 > 1292 (ACK q-27551 ‘Ack 612108 Win=65835 [ICP hpscourting--irty.com/201206Irovdesheoing-commer-networking pabloms-with-wieshark-pl-6-inding-the source-a-ratwork delays!rote “Troubleshooting Common Networking Problems with Wireshark, Pt. 6: Fining the Source of Network Delays « Courting toy HH01 0.000016 160.207.1219 160.207.1237 TCP 1292 > 4076 [SH ACK] Seq-612104 3802 0.00003 160.20712.87 160.207.12.19 TEP Reka 7ES) Win=€4SS9 Len=1192 4076 > 1292 (PSH, ACK] Seq=27551 ‘ek 613296 Win- 64343 TO? $803 0.002783 160.207.1219 160.207.12.87 TCP__CHECKSUM INCORRECT] Len=122 GET | ayouts|1083/owsbrows je TTP). 1, NTUMSSP_AUTH, User 2304 0.000144 160.207.226.48 160.207.1241 HTTP CORP\oh».D0e 1202 > 4076 [ACK| Seg£13296 3405 0.000686 160.207.1237 160,207.12.19 TCP Reen2 7673 Wina4437 Len= 1460 1292 > 4076 (PSH, ACK] Seq=614756 4076 > 1292 (ACK) seq=2767 eck 615054 Win-65535 TC? 3307 0.000018 160.207.1219 160.207.12.87 TCP__CHECKSUM INCORRECT] Len=0 Recuest calle: 4997 oarum: 39 3808 0.000032 160.207.12.19 160.207.230.326 DCIRPC etx id: 0 Looking atthe bolded Hames, we can see two things of importance Fist, the cients requested for uthent cation om the server forthe Cel, to wich esponds ma normal asian (Get, 401. Get Negotiate, 40 Challenge, Get with Credenits The interesting ting to note the speed with wich the server fesponcs fo exch step ofthis process, You will notice tht the server's 401 responses accur consistent in less than 0.001 secanés This snot consistent Wt ou” previous view of awe sever performance problem. ‘The second thing to noice s what the web Server does Inthe lst ame (3808): Rimakes an REC callto a rman contol, presumably to vey the authentication proved by she clent. Ths perhaps the most "reportant nd in the race, as shows interaction with anather server oceats before the chen is vabidled and Sven the page he has requested Now tht we have found this new piece of rformaton t's ite te determine now lng the response to this APC request takes. Ths rot easly dane by slecing te RPC request and rightdicking to choose Fallow Tce Steam’ Once this done, you can very easily see the RPC request nd response, ented oy thet Cal b we ime [Source Destination Protocol fe Request: call 4997 opm s08— ———160207:1219 160.207.2036 pcERPC 39 etna: 0 1025 > 4003 [ACK Seq7680 3654 0.255042 160.207240.36 160.207.1219 TCP Ack=1296 Win 64671 Le 8747 0.825159 160.207-250.86 160.201.1219 DCERPC Response: call: 4997 corid © “he result of ths final ters interesting. Our previous analysis showed that there was aot delay between ‘he HTTP Get and én Wed Server's resporse of 1.099 seconds (the delay Between fares 3273 and 375) ‘he araysts ofthe AC aie shows tat the delayed response from the domain contlleraccourts fo 1.08 seconds ofthat cel! Using ths information, we would be able to examine al ofthe other RPC calls 1 the DC hd verify hs finding. proving thar the domatn contol Is the primary case of the we seve! dey ‘This information's the key to Teslung he lsu, since no amount of resource allocation towards the metork Infrastructure, we server of abr series ke SQL) would actualy improve the situation a al. Bat an siddtonsl OC to handle the authentieation walle may completely resolve the tue Troubleshooting Common Networking Problems with Wireshark, TENP, atic, gedy, weshar ar trough RSS 20 You an av areata, Kaha you Oe COMMENTS (7) RELATED POSTS ‘5 by Sean Baume on Apel 72015 ~ 10:49 pm {22 by Ban lon Apr 8, 2013 = 1:0) am eal hte naoed {55 by Lnonn 09 une 17,2015 ~ 8.0 pn hpscourting-irty.com201206IrovHdesheoing-commen-networking-prabloms-with-wiceshark-p-6-inding-the source-a-ratwork delays!sar0te “Troublesnoting Common Networking Probl with Witeshark, Pt. 6: Fining the Source of Network Delays « Courting toy {24 by arian lon june 17,2013 = 816 9m ‘igae g7AKOU {55 by Unknown on ume 22,2013 = 1-48 am {20 by wiper ete on Janay 13,201 “Tanks Ba, hist ally a gra post ‘Xaualy lam wondering How you ered fame 2208? The source and destination of fame 2208 £1. by Bua il on January 13,2014 = 18 pm Name regret E-Mail required ill not be published) Wepsie ‘Submit Comment Fusion the by dstalnature | powered by Wordess Tries SS) and Comments (RSS) hpscourting-irty.com201206IrovHdesheoing-commen-networking-prabloms-with-wiceshark-p-6-inding-the source-a-ratwork delays!