Chat

:: Kelly K. Cota joined the chat.
::
:: John Fritz *Contractor* joined the chat. ::
:: Anwar M Fahad joined the chat. ::
:: Marty Hebert *Contractor* joined the chat. ::
:: Hema Pabba joined the chat. ::
:: Pedro Matheus Dias joined the chat. ::
:: mahespra@in.ibm.com - Mahesh Prasad1/India/IBM joined the chat. ::
:: Ritu Sharma joined the chat. ::
:: Yamanuri Sudheer joined the chat. ::
:: vinamboo@in.ibm.com - Vishal Namboori1/India/IBM joined the
chat. ::
:: zoharmann@us.ibm.com - Zohar Mann/Los Angeles/IBM joined the
chat. ::
:: Anuj Kumar joined the chat. ::
:: Belinda Brotherson *Contractor* joined the chat. ::
:: Venkata Rama Rao Bommireddipalli joined the chat. ::
:: grapek@us.ibm.com - Howard Grapek/Boulder/Contr/IBM joined the
chat. ::
:: mahespra@in.ibm.com - Mahesh Prasad1/India/IBM is in a meeting ::
:: zoharmann@us.ibm.com - Zohar Mann/Los Angeles/IBM is in a
meeting ::
Fritz ie Moi 3:31:0
So who do we need tro help on this? 1 AM
Anwar M Fa... 3:32:4
we need the CMS application, intel 9 AM
Fritz ie Moi 3:34:0
CMS or CEMS 2 AM
? 3:34:0
? 8 AM
for some reason data it being removed and it needs to be checked. It 6 AM
is not happening from NAS.
Fritz ie Moi 3:35:0
yes, again CMS is that a team or is it CEMs? 2 AM
CMS 3 AM
Fritz ie Moi 3:37:1
ok 8 AM
? 3:42:0
OK, we are loking for CMS in Remedy and elsewhere. 9 AM
ok 0 AM
Fritz ie Moi 4:00:3
Intel dispatcher gaveus bhavani shankar to try to help 7 AM
allright 6 AM
Fritz ie Moi 4:01:5
do we have any idea about the names of the files which appear enpty 4 AM
now?
? 4:02:0
any idea of the type of files? 2 AM
more than 2TB is gone 9 AM
? 4:02:3
i have attached the screenshot in the INC 2 AM
Fritz ie Moi 4:04:0
yes, I saw the display mentioning the CMS application 6 AM
there could have been many types of files in this qtree, It does not 4 AM
look like manual deletion. could be an application.
Fritz ie Moi 4:06:1
there could be 3 AM
INC 1750746 9 AM
? 4:08:0
The qtree was at 2160GB full and now it is at 100GB. 4 AM
Fritz ie Moi 4:09:2
can you pull any names that may help locate who uses this space or 8 AM
what the application might be?
I am trying to pull information on all the CMS qtrees i can 3 AM
? 4:18:4
security can give us better information 1 AM
:: Ramya Yalamanchili joined the chat. ::
:: Satish Jain joined the chat. ::
Bhavani And... 4:21:1
Hi all 8 AM
Satish Jain 4:21:2
Hi 2 AM
Fritz ie Moi 4:22:2
Satish the thought for netback up is to check to see if a backup or 3 AM
restore has caused this.
Satish Jain 4:23:4
could u give me server name 0 AM
Fritz ie Moi 4:24:2
he share path is \\erdhfdnas002\HFD_flx_306$\CMSData 0 AM
? 4:26:4
Anwar, have you engaged anyone from NAS engineering? 3 AM
NetApp, vendor? 0 AM
? 4:28:1
I have checked the filer and nothing is wrong with it, no errors. 4 AM
Fritz ie Moi 4:29:1
Anwar, Bhavani is from Intel please tell him what you want him to 6 AM
check?
? 4:34:4
Anwar or Ritu I got an intel peson here please tell them what you want 3 AM
checked now.
Yes john, Bhavani is looking at the screenshot attachent in the INC 5 AM
Fritz ie Moi 4:36:0
Anwar, the service Manager wants to know if you have engaged NAS 5 AM
engineering.
? 4:36:1
? 1 AM
Fritz ie Moi 4:42:2
Anwar? 9 AM
Satish Jain 4:42:5
We did not perform any restore..on this server...so restore is not the 7 AM
cause
I have not engaged the vendor yet, I am opening a case with netapp 7 AM
right now.
Fritz ie Moi 4:44:0
Anwar, I did not ask if you engaged a vendor; I asked if you engaged 3 AM
NAS engineering?
? 4:45:0
Bhavani, can you post wat you are working on for this? 3 AM
? 4:45:1
Bhavani, can you post what you are working on for this? 9 AM
John I am from NAS 9 AM
I am not find anything, checking with my team are they done any 0 AM
home driver migraction
Fritz ie Moi 4:47:1
Anwar, are you NAS engineering? If not have you engaged them? 7 AM
? 4:48:1
This is an "all hands on deck" situation 2 AM
Yes I am from NAS engineering 7 AM
? 4:50:0
There only one team for NAS Storage 7 AM
Fritz ie Moi 4:51:0
Do you have US support now? If not we need them, now. 2 AM
? 4:53:2
We need you to enfgge them are you? 6 AM
I am checking with Kelly 5 AM
Fritz ie Moi 4:53:4
Kelly who? 7 AM
Kelly Cota 0 AM
Fritz ie Moi 4:54:4
Do you have her on the phone? 7 AM
John, I trying to get Marty Hebert here. 2 AM
? 5:04:0
I cant get to him, have left voice messages. 0 AM
Fritz ie Moi 5:06:2
Did you speak to Kelly? 5 AM
Ritu Sharma... yes 5:07:4
john i 5 AM
spoke
to kelly
she is
availab
le
Fritz ie Moi 5:08:4
Did you try Marty cell phone? 4 AM
Ritu Sharma... yes i 5:09:1
did and 4 AM
tried to
contact
daniel
too left
a vm
Fritz ie Moi 5:09:4
Maerty on Mobile: 1-972-533-2716 ? 0 AM
Ritu Sharma... he is 5:10:0
availab 8 AM
le now
i think
kelly
able to
rech
him
Marty Heber... 5:10:2
2 AM
:: Kelly K. Cota joined the chat. ::
Kelly K. Cota 5:10:3
hi 8 AM
Hello, trying to get into my vpn 3 AM
did Venkata join also? 5 AM
? 5:10:5
please invite 4 AM
Ritu Sharma... sure i 5:11:1
m doin 6 AM
it
Fritz ie Moi 5:12:0
Marty please join a call on this 1-866-365-4406 pc 0225524, thank 4 AM
you
:: Ritu Sharma left the chat. ::
:: Yamanuri Sudheer left the chat. ::
Kelly joined 0 AM
? 5:20:3
Ritu and Anwar, per the sev 1 critical incident process you need to 4 AM
involve your first line mgmt if the issue is not resolved within 1 hr, do
you remember this process the storage team is to follow?
Ritu Sharma... i tried 5:21:0
to call 6 AM
him
but he
is not
reacha
ble
did you go to his manager 4 AM
I Checked with Team and found there is No Data Migration Activity is 7 AM
Not Performed by the Intel Team from Past 2 Days
please call Mahesh Prasad as the second line and I will call Debi to 0 AM
update her
Ritu Sharma... k i m 5:26:1
doin it 7 AM
allright 8 AM
impact to customer is ? 7 AM
Fritz ie Moi 5:32:5
Kelly, I believe that is unknown as of yet. 8 AM
:: Vishal Namboori joined the chat. ::
who is opening the yellow event 8 AM
Fritz ie Moi 5:33:3
7 AM
? 5:33:4
corinne lawrence 4 AM
? 5:34:0
hartford Service Manager 5 AM
so how can we have a sev1 incident and not know what the impact to 2 AM
our customer is
:: Vishal Namboori left the chat. ::
Fritz ie Moi 5:36:3
NAS concern that we had 2.11 TB of data go to 100MB of data and do 9 AM
not kow if that is a thing that should have happened or not. EITSM and
Service Manager have not been able to access the impact or not so
they're erring on the side of caution.
? 5:37:2
The other piece is they still see the file structures just all 0 bytes 9 AM
do we know who owns the data and has anyone called them? 8 AM
Fritz ie Moi 5:41:4

no we do not know. The one query that NAS ran through DFM showed 4 AM
the CMS application had 2.11 TB with an SA name Rick Sevigny who is
gone from IBM
? 5:42:2
now getting localized to the nested share not the whole Q tree 5 AM
? 5:42:4
per Marty 1 AM
has anyone lookied into change mgmt to see if something was done 6 AM
inadertently?
Fritz ie Moi 5:44:2
SD has checked and found nothing that seems tied to this 5 AM
Venkata Ra... 5:51:0
i am on 0 AM
:: Vishal Namboori joined the chat. ::
Ritu, can you give Zohar a call for another set of eyes 8 AM
Ritu Sharma... ok i m 5:52:1
doin it 3 AM

I totally agree Marty, lets get an architect to review before we make 9 AM
any decision and not waste any more time
:: mahespra@in.ibm.com - Mahesh Prasad1/India/IBM joined the chat. ::
:: Hema Pabba left the chat. ::
Ritu Sharma... zohar 5:55:1
is not 5 AM
pickin
left a
vm
call an alternate number for him if you have one 7 AM
? 5:56:0
if you don't then need to engage his manager STeve poss 6 AM
Ritu Sharma... i dont 5:57:5
have 3 AM
his
alterna
te no
called
him
twice
not
availab
le
call steve poss then 5 AM
Ritu Sharma... k callin 5:59:0
him 8 AM
Venkata Ra... 5:59:1
Anwar..Can u engage Steve Poss 1 AM
if he can't reach Zohar he can get someone else from his team 5 AM
perhaps
Ritu Sharma... he is 5:59:4
not 2 AM
availab
le too
Fritz ie Moi 6:00:3
Ritu did you try his mobile? 6 AM
Phone: 1-630-907-0658
Mobile: 1-630-215-5856
? 6:00:5
Steve Poss 3 AM
Ritu Sharma... i tried 6:01:1
his 2 AM
phone
lemme
try his
mobile
?
i talked 6:06:3
to 7 AM
steve
he is
calling
zohar
to get
online
he
would
update
kelly
on the
same
Is it possible to verify how long it takes for the data to go back to 0? 3 AM
My small test case is still showing data.
? 6:20:0
2 AM
:: Zohar Mann joined the chat. ::
marty is calling netapp right now Zohar 0 AM
Zohar Mann 6:26:5
can someone post the number here pls? 0 AM
Fritz ie Moi 6:26:5
Tech conf call opening 1-866-423-8350 Participant Passcode: 4 AM
4210624480
? 6:27:3
line is open now 9 AM
zohar per our crit sit process this has been going on since 9:21 last 7 AM
night so that is why you are now engaged to lend another set of eyes
Fritz ie Moi 6:45:0
Anwar, are you joining the tech call? We moved to 1-866-423-8350 2 AM
Participant Passcode: 4210624480
Joining 5 AM
Fritz ie Moi 6:47:3
OK, we are awating MArty's return from engageing Netapps 9 AM
? 6:49:0
Marty what file did you restore? We can have it checked to see if it still 2 AM
has bytes associated with it.
Zohar Mann 6:49:2
want to make sure we don't have more than 1 nas admin making 2 AM
changes- Is Marty on point or someone from steady state team right
now?
Marty only 7 AM
Zohar Mann 6:49:4
k thx 2 AM
Fritz ie Moi 6:49:5
super 0 AM
\\erdhfdnas002\cmsdata$\CMSdata\ShrData\!Drive 8 AM
Failure\Alex\ALEX_TEST_FILES\M_Etes
? 6:52:5
this folder and it is still showing data 1 AM
? 6:52:5
meaning not 0 4 AM
Fritz ie Moi 6:53:2
thank you 1 AM
? 6:53:3
When did that get restored? 3 AM
before I left the call 1 AM
Zohar Mann 6:53:5
so based on DFM, it looks like capacity went way down after 10PM last 4 AM
night
? 6:54:2
10PM ET 2 AM
2160GB to a 100GB on the qtree 2 AM
Zohar Mann 6:57:3
7/28/2010 9:45 2,318,798,311,424.00 4 AM
7/28/2010 20:45 726,029,418,496.00
7/28/2010 21:24 -
? 7:00:4
8 AM

Zohar Mann 7:13:3
I see over 50 nested shares- any idea if problem is isolated to specific 6 AM
shares?
Fritz ie Moi 7:17:1
Zohar I thought Marty got it 7 AM
erdhfdnas002/cmsdata$/CMSdata/ShrData
:: Satish Jain left the chat. ::
Zohar Mann 7:21:3
Has the security team been engaged? 7 AM
brb 3 AM
Fritz ie Moi 7:22:0
yes Zohar 4 AM
All, my test restore is still OK. Should be try some more restores while 6 AM
we try and figure out what happened
Fritz ie Moi 7:22:3
Security is checking s add or deletion or no loger employed could have 9 AM
effected this.
? 7:28:4
thanks Marty 1 AM
? 7:29:1
Marty, did you quantify this to 46G + or - ? 1 AM
46G you talking size ?? 0 AM
Fritz ie Moi 7:36:2
yes 4 AM
should we try the restore that the customer tried and see if it holds? 7 AM
:: Venkata Rama Rao Bommireddipalli left the chat. ::
That would be a great idea. Can we get the path? 7 AM
Zohar Mann 7:47:1
CMSshrdata /vol/HFD_flx_306/CMSdata/cmsdata/ShrData CMS 9 AM
ShrData
AD1\CBG_CMSShrdata / Change
AD1\CLShrapps Admins / Change
AD1\CMS_CMSShrData / Change
AD1\cmsPerfAnalytics / Full Control
AD1\CMSShrData / Change
ERDHFDNAS002\CMSshrdataaccess / Change
AD1\CMSSpectrum Approver / Change
ERDHFDNAS002\CMSusers / Change
AD1\COR_CMSSHRDATA / Change
AD1\ECCC Admins / Full Control
everyone / Change
AD1\PLC_CMSShrData / Change
AD1\RSD-ATS-ERDHFDNAS002-CMSShrData-RO / Read
AD1\RSD-ATS-ERDHFDNAS002-CMSShrData-RW / Change
AD1\RSD-Claims-ERDHFDNAS002-CMSShrData-ECM-RW / Change
AD1\RSD-ECM-ECM-RW / Change
AD1\RSD-ERDHFDNAS002-CMSShrData-UCM-RO / Read
AD1\RSD-ERDHFDNAS002-CMSShrData-UCM-RW / Change
AD1\RSD-IM-AD1-Prod-HIG-Corp-RSD-CMSShrData-Warehouse-RW /
Change
AD1\XXFTP035 / Read
\\erdhfdnas002\cmsshrdata\CLCS Commission Project\2010 ISD 5 AM
Playbooks
Zohar Mann 8:13:2
It would be good to announce if customer is on call (or this chat via 3 AM
shared screen).
Fritz ie Moi 8:19:0
I do not think we hae cutomers on the tech call 7 AM
? 8:19:3
Zohan, I do not think we have customers on the tech call 4 AM
Zohar Mann 8:19:3
Mike was on listening with Bridget I think? 9 AM
Fritz ie Moi 8:20:0
Ah, ha mentioned Kathy I missed Bridget 3 AM
bbrotherson... 8:20:0
who from this chat room in on the technical bridge? 3 AM
Zohar Mann 8:20:1
I am 2 AM
I am 4 AM
I am 0 AM
Zohar Mann 8:20:2
15 callers on the bridge when I rejoined 8 AM
Fritz ie Moi 8:20:3
I'm checking with Mike on Bridget 2 AM
when i get a chance, I'm going to do a roll call on the bridge...dont' 9 AM
want to interrupt the current flow though...
? 8:21:1
Zohan, did Jim join after you? 4 AM
Fritz ie Moi 8:21:2
yes 0 AM
? 8:21:2
BB 2 AM
ok...so that would be about 16 0 AM
Fritz ie Moi 8:24:3
Bridget is not on the call any longer 5 AM
dropping from the call 4 AM
ok 9 AM
Fritz ie Moi 8:30:2
thank you Kelly 8 AM
sure I will check back in shortly 9 AM
? 8:30:5
on another urgent call 2 AM
Fritz ie Moi 8:31:0
np! 1 AM
backup team? who is still here? 0 AM
Yamanuri Su... 8:34:5
iam there from backup team 4 AM
vishal is there from restore ops team 1 AM
? 8:35:4
do you need any assistance from backup team 7 AM
the customer is asking about the possibility of a restore 9 AM
? 8:36:2
would you please join the conf call? 1 AM
iam in home 1 AM
1-866-423-8350 Participant Passcode: 4210624480 4 AM
? 8:36:3
ok 6 AM
? 8:36:4
is there someone that could patch you? 4 AM
i willl ask anuj to join the call 4 AM
ok 6 AM
? 8:36:4
that will work 9 AM
both are in meeting 0 AM
? 8:37:1
once meeting gets completed then let you know 2 AM
? 8:37:3
please provide me the details so that i will check from backup end 2 AM
? 8:38:2
hi anuj 4 AM
? 8:38:3
are you there? 3 AM
? 8:38:4
please join the call 8 AM
? 8:39:0
Anuj Kumar 8:47:4
Hi Sudheer,, i am joining it after cadence call 7 AM
? 8:48:0
i will finish it in next 15 min 1 AM
ok anuj 8 AM
? 8:48:2
i asked vishal to check 4 AM
:: Pedro Matheus Dias joined the chat. ::
:: Howard (Howie) Grapek *Contractor* joined the chat. ::
Hi John 6 AM
welcome Pedro 9 AM
:: Srinivas R Baddam joined the chat. ::
Yamanuri ... 6:21:3
Manish from backup team will join the chat 5 PM
Pedro Mat... 6:21:3
8 PM
Belinda.. good morning
? 6:21:4
Good morning everyone 1 PM
Belinda Br... 6:21:4
good morning 8 PM
? 6:22:0
i need a restore resource on the conference call 8 PM
Pedro Mat... 6:22:2
no one has joined yet? 3 PM
Yamanuri ... 6:22:2
yes 3 PM
:: Manish H Ghildyal joined the chat. ::
Yamanuri ... 6:22:3
i asked vishal to join the call 2 PM
:: Venkatesh Pillalamarri *Contractor* joined the chat. ::
let me check... i just had a "ding" on the call 8 PM
? 6:23:0
he is not answering on the call yet 4 PM
grapek@u... 6:23:0
pedro will join at 9.. and I'm asking vinny to join now from the US to 5 PM
help understand the issue and help drive the global resources.
Yamanuri ... 6:23:3
hi manish 7 PM
? 6:23:4
please join the call 1 PM
? 6:23:5
1-866-423-8350 Participant Passcode: 4210624480 3 PM
? 6:23:5
i updated to vishal 9 PM
:: vinamboo@in.ibm.com - Vishal Namboori1/India/IBM is available ::
Yamanuri ... 6:24:0
he will give you details 7 PM
:: jamartin@us.ibm.com - Jo-Ann Martin/Southbury/IBM joined the
chat. ::
:: jamartin@us.ibm.com - Jo-Ann Martin/Southbury/IBM is available ::
Pedro Mat... 6:25:5
Vishal is joining now 9 PM
:: Hema Pabba is away ::
:: Hema Pabba is available ::
:: Bhavani S Ande joined the chat. ::
:: John Fritz *Contractor* left the chat. ::
:: Kelly K. Cota is away ::
:: Kelly K. Cota is available ::
:: morinmg@us.ibm.com - Michael G Morin/Poughkeepsie/IBM joined
the chat. ::
:: morinmg@us.ibm.com - Michael G Morin/Poughkeepsie/IBM is
available ::
:: morinmg@us.ibm.com - Michael G Morin/Poughkeepsie/IBM left the
chat. ::
offline ::
:: rajyadav@in.ibm.com - Raju Yadav/India/IBM joined the chat. ::
:: rajyadav@in.ibm.com - Raju Yadav/India/IBM is in a meeting ::
the chat. ::
available ::
:: Bhavani S Ande left the chat. ::
:: Bhavani S Ande is offline ::
Vinnie has stated on the call that the last full backup was from the 2 PM
10th... there are incrementals...but
? 6:41:0
so we need to track that as an follow-up item as to why this full 5 PM
backup hasn't been successful lately
Venkatesh... 6:46:3
Most recent attempted FULL was on 25 4 PM
who has Domain Admin level access that can copy these files? 9 PM
Venkatesh... 6:47:2
Backed up 486,879,232 in 4.53 hours and then went for a tape 0 PM
change. The tape change resulted in a bad tape
ouch 6 PM
Venkatesh... 6:47:5
the failed backup not subsequently remediated 0 PM
who has Domain Admin level access that can copy / restore these 0 PM
files?
:: rajyadav@in.ibm.com - Raju Yadav/India/IBM is away ::
:: rajyadav@in.ibm.com - Raju Yadav/India/IBM is in a meeting ::
morinmg@... 7:03:0
why wouldnt we have the customers do their own restores as they 4 PM
identify the file is missing?
? 7:03:1
i thought we said we have issue because we do not have permissions? 7 PM
? 7:03:2
did i miss something 3 PM
i don't think so... i don't completely understand who has the 2 PM
permissions so my question is to the general audience to confirm that
i'm not missing something
morinmg@... 7:04:0
right 7 PM
:: John Fritz *Contractor* is away ::
:: Ritu Sharma is away ::
:: jamartin@us.ibm.com - Jo-Ann Martin/Southbury/IBM is away ::
zoharman... 7:17:0
permissions: everyone 9 PM
is that as it should be? 3 PM
zoharman... 7:17:3
CMSshrdata /vol/HFD_flx_306/CMSdata/cmsdata/ShrData CMS 4 PM
ShrData
AD1\CBG_CMSShrdata / Change
AD1\CLShrapps Admins / Change
AD1\CMS_CMSShrData / Change
AD1\cmsPerfAnalytics / Full Control
AD1\CMSShrData / Change
ERDHFDNAS002\CMSshrdataaccess / Change
AD1\CMSSpectrum Approver / Change
ERDHFDNAS002\CMSusers / Change
AD1\COR_CMSSHRDATA / Change
AD1\ECCC Admins / Full Control
everyone / Change
AD1\PLC_CMSShrData / Change
AD1\RSD-ATS-ERDHFDNAS002-CMSShrData-RO / Read
AD1\RSD-ATS-ERDHFDNAS002-CMSShrData-RW / Change
AD1\RSD-Claims-ERDHFDNAS002-CMSShrData-ECM-RW / Change
AD1\RSD-ECM-ECM-RW / Change
AD1\RSD-ERDHFDNAS002-CMSShrData-UCM-RO / Read
AD1\RSD-ERDHFDNAS002-CMSShrData-UCM-RW / Change
AD1\RSD-IM-AD1-Prod-HIG-Corp-RSD-CMSShrData-Warehouse-RW /
Change
AD1\XXFTP035 / Read
? 7:18:0
top line is the share. lines under it are the user or group and their 0 PM
access priv
Pedro Mat... 7:20:0
(will be back soon 3 PM
:: Anwar M Fahad is away ::
chat. ::
:: Hema Pabba left the chat. ::
:: Pedro Matheus Dias is away ::
:: Ritu Sharma is available ::
:: Pedro Matheus Dias is available ::
:: Yamanuri Sudheer left the chat. ::
:: Gerald L. Gase joined the chat. ::
any updates? 4 PM
Gerald L. G... 8:08:0
Jim Hoover (Hartford Info Security) is investigating logs to detect 9 PM
anything from an ID perspective.
good 9 PM
the chat. ::
Gerald L. G... 8:09:1
Snap Mirror is at 100GB as of 10:26 (45 minutes for 100GB). 0 PM
available ::
and we are in the middle of a volume snapmirror to another aggregate 6 PM
that has free space.
? 8:09:3
how much data has been transferred 6 PM
? 8:10:1
want to estimate the completion time 9 PM
? 8:10:4
in the meantime assume someone is working with the customer to get 8 PM
a prioritized list for the restore of what is first?
? 8:10:5
or has a decision been made on how to do that yet? 9 PM
zoharman... 8:13:4
I think the restore plan is still pending unless they worked it out on the 3 PM
call while I was away.
:: morinmg@us.ibm.com - Michael G Morin/Poughkeepsie/IBM is away ::
:: Venkata Rama Rao Bommireddipalli is offline ::
I am only in this st chat not on the call 9 PM
Gerald L. G... 8:16:0
Sorry Kelly. I am listening to updates from Jim Hoover and others. 0 PM
Doesn't appear to be anything from an individual user event nor any
virus scan.
:: Anwar M Fahad is available ::
ok thats good 6 PM
Gerald L. G... 8:18:3
Good for that piece. But still leaves question as to what happened. 5 PM
no one found a script yet, did we hear back from Netapp on the logs? 0 PM
Gerald L. G... 8:20:4
Marty did indicate Symptoms look like something that could be caused 1 PM
archiving but not conclusive
I know they are huge 2 PM
Gerald L. G... 8:21:0
Marty just stepped away from the call to ask. 3 PM
and we ruled out permission problems right? 7 PM
Gerald L. G... 8:23:1
Not sure. Belinda, was that ruled out earlier? It wasn't mentioned in 3 PM
the during the last 15 minutes.
zoharman... 8:23:5
Did I hear correctly earlier that a tape restore was requested for this 6 PM
qtree yesterday?
i am not aware of a previous restore request 7 PM
zoharman... 8:33:1
what was the name of the tape backup resource that was on the call 8 PM
before?
Vinnie 6 PM
Ventatesh Pillalamari aka. Vinnie 1 PM
:: Manish H Ghildyal is away ::
grapek@u... 8:37:3
im here too. and on phone with vinny, whats up? 5 PM
? 8:38:0
oh, i see the question - i'll check in a couple of minutes. 3 PM
:: Kelly K. Cota left the chat. ::
zoharman... 8:38:1
did vinnie say that a backup or a restore attempted yesterday at 8 PM
~1300?
:: Manish H Ghildyal is available ::
zoharman... 8:40:5
Belinda- have you created a timeline yet for the isssue? 6 PM
vinamboo... 8:41:0
Restore attempted 13:06:57 1 PM
i am working on a timeline for the investigation 0 PM
? 8:41:3
i have not started the overall event timeline 6 PM
? 8:42:2
this issue started around 17:05 yesterday 9 PM
zoharman... 8:42:3
but the data deletion started earlier 8 PM
the INC was opened at 00:57 this morning 4 PM
? 8:43:1
the first I have heard of issues was from 5:05 pm ET yesterday 9 PM
evening
zoharman... 8:44:1
we need to refine the time but based on the vague graph I pasted, the 9 PM
deletions appear to perhaps have started around 12 noon yesterday
(07/28). Maybe a little earlier. reserach required.
7/28/2010 - 5:04 pm is when the data started tp disappear... some 0 PM
disappeared closer to 7:00 pm
? 8:44:4
oh really? 1 PM
? 8:45:1
Zohar: are you referring to 12 noon ET? 3 PM
available ::
:: rajyadav@in.ibm.com - Raju Yadav/India/IBM left the chat. ::
zoharman... 8:54:0
yes. trying to get a more granular report now 0 PM
:: mahespra@in.ibm.com - Mahesh Prasad1/India/IBM left the chat. ::
Kishore is now on the call as well 8 PM
? 9:00:5
we are discussing the best method for copying the data back to its 2 PM
original location
:: Kishore Guduru joined the chat. ::
:: Kishore Guduru is in a meeting ::
zoharman... 9:16:1
These times are ET: 5 PM
? 9:16:3
7/28/2010 10:00 2,318,798,311,424 1 PM
7/28/2010 22:00 726,029,418,496
7/29/2010 8:00 117,892,698,112

is that the total file size you have listed? 1 PM
zoharman... 9:17:5
that was the qtee used capacity in bytes 9 PM
Venkatesh... 9:18:2
Regarding the restore question - 5 PM
? 9:18:3
It started at 07/28/10 14:57:56 0 PM
? 9:18:5
Appeared like it hung for a long time 1 PM
? 9:18:5
Then finally the failure of that restore: 9 PM
? 9:19:0
07/29/10 01:06:00 - Error ndmpagent(pid=2070) aborting operation - 0 PM
no mover progress
? 9:19:3
The data that was attempted for restore was: 4 PM
? 9:19:4
/vol/HFD_flx_306/CMSdata/CMSdata/ShrData/ on erdhfdnas002 6 PM
what is the size now? 4 PM
? 9:20:2
has it decreased again since 8am? 2 PM
Vinnie, why did we attempt a restore? 6 PM
? 9:26:2
Zohar: what is the current size of the directory? 1 PM
zoharman... 9:26:5
Have to get that from the NAS Admin- I'm not logged into the filer, 8 PM
just a reporting tool (non realtime).
ok 8 PM
? 9:27:1
i can ask then...i thought you may have it handy 4 PM
zoharman... 9:27:4
but I'm very curious about the restore attempt 2 PM
Howie... can you join the call for a few? 1 PM
? 9:28:0
and Zohar, you might be interested as well 7 PM
zoharman... 9:28:1
k 2 PM
? 9:28:3
I'm going to try my landline- again pls tell me if noisy 1 PM
Venkatesh... 9:28:4
Somebody from the global restore team needs to answer that 7 PM
question
sure 8 PM
? 9:31:3
Howie... can you join? 4 PM
? 9:32:0
who is still here from Restore team? 1 PM
:: vinamboo@in.ibm.com - Vishal Namboori1/India/IBM left the chat. ::
chat. ::
vinamboo... 9:33:0
This is Vishal from Restore Team 4 PM
IMAGE [smile] 0 PM
? 9:33:2
Vishal, can you please join the bridge line? 2 PM
? 9:33:3
need more information on the attempted restore from yesterday 8 PM
? 9:34:4
NAS team... please confirm the current size of the directory. 3 PM
vinamboo... 9:35:2
joined the call 6 PM
thank you for joining Vishal 0 PM
? 9:36:0
Vishal, do you know how requested the restore from yesterday? 9 PM
Venkatesh... 9:38:1
Vishal, the restore is P&C jobid 1974868, to 0 PM
erdhfdnas002:/vol/HFD_flx_306/CMSdata/CMSdata/ShrData/ queued
at 07/28/10 13:06:59, but actually went active at 07/28/10 14:57:56
is there an INC? 7 PM
vinamboo... 9:38:3
yes 2 PM
can you provide the INC from the restore request 5 PM
vinamboo... 9:38:4
INC1747899 8 PM
thank you...looking it up now 4 PM
vinamboo... 9:39:5
INC1739065 this is the correct one 4 PM
ok 2 PM
? 9:41:1
so you were trying to restore the 2009 PL Forum folder? 6 PM
vinamboo... 9:41:2
yes 4 PM
did you restore the whole path 9 PM
erdhfdnas002/vol/HFD_flx_306/CMSData/ShrData?
vinamboo... 9:43:2
I need to check with my colleague who initiated the restore: give me 5 2 PM
min
ok 6 PM
? 9:43:4
we need information on where the restore error'd out as well 0 PM
vinamboo... 9:43:4
ok 7 PM
what is the current size of the NAS share directory? 0 PM
? 9:44:4
have we lost any more data since 08:00 am ET? 3 PM
? 9:44:5
I need to determine if the system is now stable 1 PM
chat. ::
zoharman... 9:52:1
Marty- can you please check /etc/log/backup to see if it had restore 6 PM
data?
:: sreekanthj@in.ibm.com - Sreekanth Jayaprakash/India/IBM joined the
chat. ::
:: sreekanthj@in.ibm.com - Sreekanth Jayaprakash/India/IBM is
available ::
Belinda Br... 10:02:
Vishal, who kicked off the restore? 18 PM
we need to understand what steps were taken to ensure we are taking 44 PM
the correct restore actions
:: Anwar M Fahad left the chat. ::
:: Anwar M Fahad is offline ::
:: Shivaram Aswathaiah joined the chat. ::
Marty, can you tell me current size of share on the NAS? 50 PM
Marty Heb... 10:08:
Here are all the impacted Shares in this QTree 58 PM
CMSdata$ /vol/HFD_flx_306/CMSdata
CMSsoftware$ /vol/HFD_flx_306/CMSdata/cmssoftware
CMSelecmrkt /vol/HFD_flx_306/CMSdata/cmsdata/elecmrkt
CMSwinapps /vol/HFD_flx_306/CMSdata/cmssoftware/winapps
CMSshrapps /vol/HFD_flx_306/CMSdata/cmsdata/shrapps
CMSabr$ /vol/HFD_flx_306/CMSdata/cmsdata/abr
CMSmassauto /vol/HFD_flx_306/CMSdata/cmsdata/shrapps/massauto
CMSEndorsement$ /vol/HFD_flx_306/CMSdata/Cmsdata/Endorsement
Study
CMScontrollers$ /vol/HFD_flx_306/CMSdata/cmsdata/Controllers-RO
Controllers-RO
CMSlars /vol/HFD_flx_306/CMSdata/cmsdata/shrData/Loss
Control/Lars Loss Control LARS
CMSmicrocenter
/vol/HFD_flx_306/CMSdata/cmsdata/shrData/Microcenter Local
Presence Microcenter Local Presence
CMSmiddle_market
/vol/HFD_flx_306/CMSdata/cmsdata/shrData/Middle Market Account
Score Middle Market Account Score
CMSfrmtest$ /vol/HFD_flx_306/CMSdata/cmsdata/formtest CMS
Product Registry
cmsteradata$ /vol/HFD_flx_306/CMSdata/cmssoftware/teradata
Teradata software location
CMSprtword$ /vol/HFD_flx_306/CMSdata/cmsdata/prtword CMS
Product Registry
CMSfrmprod$ /vol/HFD_flx_306/CMSdata/cmsdata/formprod CMS
Product Registry
cmsstforms$ /vol/HFD_flx_306/CMSdata/cmsdata/stateforms CMS
Product Registry
CMSprlword$ /vol/HFD_flx_306/CMSdata/cmsdata/prlword CMS
Product Registry
CMSmrkting /vol/HFD_flx_306/CMSdata/cmsdata/Mrkting CMS
Marketing Share
CMSbkfsoftwr
/vol/HFD_flx_306/CMSdata/cmsdata/ShrData/warehouse/software BKF
Software for FTP
CMSshrdata /vol/HFD_flx_306/CMSdata/cmsdata/ShrData CMS
ShrData
CMSscicp /vol/HFD_flx_306/CMSdata/CMSdata/reference/scic
paperless CMS SCIC Paperless (admin share)
EDW Admin /vol/HFD_flx_306/CMSdata/cmsdata/ShrData/EDW Admin
EDW /vol/HFD_flx_306/CMSdata/cmsdata/ShrData/EDW
CMSLCAIT /vol/HFD_flx_306/CMSdata/cmsdata/shrData/CMSLCAIT
CMSPLForms /vol/HFD_flx_306/CMSdata/cmsdata/shrapps/pl_forms
CMSreference /vol/HFD_flx_306/CMSdata/CMSdata/reference CMS
Reference
CMSncciw /vol/HFD_flx_306/CMSdata/cmsdata/shrapps/ncci
worksheets
cs_Prd$ /vol/HFD_flx_306/CMSdata/cmsdata/shrdata/cs_Prd
cs_Ctst$ /vol/HFD_flx_306/CMSdata/cmsdata/shrdata/cs_Ctst
cs_Dev$ /vol/HFD_flx_306/CMSdata/cmsdata/shrdata/cs_Dev
cs_Trn$ /vol/HFD_flx_306/CMSdata/cmsdata/shrdata/cs_Trn
cs_Pspt$ /vol/HFD_flx_306/CMSdata/cmsdata/shrdata/cs_Pspt
cs_Ctst2$ /vol/HFD_flx_306/CMSdata/cmsdata/shrdata/cs_Ctst2
cs_Dev2$ /vol/HFD_flx_306/CMSdata/cmsdata/shrdata/cs_Dev2
CMStraining /vol/HFD_flx_306/CMSdata/cmsdata/Training Materials
CMS Training
HR /vol/HFD_flx_306/CMSdata/cmsdata/ShrData/HR
scorecard$ /vol/HFD_flx_306/CMSdata/CMSdata/ShrData/ucm hidden
share for manage scorecard folder
CMS20052007lmp
/vol/HFD_flx_306/CMSdata/cmsdata/ShrData/2005-2007 Local Market
Plans
CMSlmp /vol/HFD_flx_306/CMSdata/cmsdata/ShrData/Local Market
Planning
IFORMS_VALIDATION
/vol/HFD_flx_306/CMSdata/cmsdata/shrapps/IFORMS_VALIDATION
CFO Finance /vol/HFD_flx_306/CMSdata/cmsdata/ShrData/CFO
Finance
Third Party Pricing Model (TPPM)
/vol/HFD_flx_306/CMSdata/cmsdata/shrapps/Third Party Pricing Model
(TPPM)
CMSal /vol/HFD_flx_306/CMSdata/cmsdata/shrdata/agency licensing &
contracting
CFO$ /vol/HFD_flx_306/CMSdata/cmsdata/shrdata/cfo Share for Mary
Beth Miller
DFSANCHOR-SmartCard$
/vol/HFD_flx_306/CMSdata/cmsdata/ShrData/SmartCard
DFSANCHOR-Warehouse$
/vol/HFD_flx_306/CMSdata/cmsdata/ShrData/Warehouse
DFSANCHOR-BKFITSupport$
/vol/HFD_flx_306/CMSdata/cmsdata/ShrData/BKF IT Support
DFSANCHOR-MicrosoftVisualStudio$
/vol/HFD_flx_306/CMSdata/cmsdata/ShrData/Microsoft Visual Studio
StaffingCoordinators
/vol/HFD_flx_306/CMSdata/cmsdata/ShrData/Staffing Coordinators
CHERYL_from actuarial
/vol/HFD_flx_306/CMSdata/cmsdata/ShrData/CHERYL_from actuarial
Earley /vol/HFD_flx_306/CMSdata/cmsdata/ShrData/ECM/Earley
Strategic Initiatives Folder
/vol/HFD_flx_306/CMSdata/cmsdata/ShrData/Strategic initiatives
Strategic Initiatives Folder
DFSAnchor_eBT_Rpt_DB$
/vol/HFD_flx_306/CMSdata/CMSdata/ShrData/eBT_Rpt_DB eBT
Reporting Database - Harris, Chuck (Comm Lines, IT)
DFS_Anchor_CMSBKFSoftwr$
/vol/HFD_flx_306/CMSdata/cmsdata/ShrData/warehouse/software BKF
For FTP
? 10:10:
Belinda, 10 PM
it is kind of misleading due to the snaps eating into the real space.
bash-2.05$ rsh erdhfdnas002 df -V -g HFD_flx_306

Filesystem total used avail capacity Mounted on
/vol/HFD_flx_306/ 2239GB 1915GB 323GB 86% /vol/HFD_flx_306/
/vol/HFD_flx_306/.snapshot 248GB 2047GB 0GB 823%
/vol/HFD_flx_306/.snapshot
? 10:10:
The snaps are 2047 GB 27 PM
? 10:10:
much larger than the 248 GB reserved 43 PM
trying to determine if we are still loosing data...from this 57 PM
status...7/28/2010 10:00 2,318,798,311,424
7/28/2010 22:00 726,029,418,496
7/29/2010 8:00 117,892,698,112
? 10:11:
have we lost any more after 8am? 08 PM
:: Venkatesh Pillalamarri *Contractor* left the chat. ::
Marty Heb... 10:14:
I do not think we still are. My numbers have not changed in the past 3 37 PM
or 4 hours.
Marty Heb... 10:15:
And data that has been restored since 5 AM EST is still there 15 PM
:: Anuj Kumar left the chat. ::
Marty, can you please rejoin the bridge 17 PM
Marty Heb... 10:18:
I did not know I left 26 PM
? 10:18:
ok 27 PM
have HIG mgmt on the call for status... Dan S at this point 31 PM
? 10:18:
Bridget may be joining soon too 48 PM
vinamboo... 10:29:
Belinda ,Shivram joined from the Restore team:I'm logging out for the 27 PM
day.
did you speak to who started the restore? 55 PM
vinamboo... 10:30:
Shivram worked on this: 08 PM
? 10:30:
He will update you. 15 PM
Shivaram ... 10:30:
Hi 16 PM
:: grapek@us.ibm.com - Howard Grapek/Boulder/Contr/IBM is away ::
:: Venkata Rama Rao Bommireddipalli is away ::
:: sreekanthj@in.ibm.com - Sreekanth Jayaprakash/India/IBM left the
chat. ::
chat. ::
:: Shivaram Aswathaiah is away ::
zoharman... 10:50:
Hi Belinda 01 PM
? 10:50:
Are you sending out status updates for mgmt? 12 PM
:: Shivaram Aswathaiah is available ::
:: Avinash Gautam joined the chat. ::
zoharman... 10:54:
Hello. Who is sending out the status updates for mgmt for this issue? 59 PM
chat. ::
chat. ::
available ::
i will prepare a status and send soon 29 PM
:: John Fritz *Contractor* is available ::
zoharman... 10:58:
thx 09 PM
:: Manish H Ghildyal is away ::
:: Manish H Ghildyal is available ::
Kelly K. Cota Belinda 11:07:
please 23 PM
include
myself
?
and 11:07:
Debi 29 PM
ralston
too
:: Sheelnidhi Gupta joined the chat. ::
ok 17 PM
:: Frank D. (Frank) Breen joined the chat. ::
:: Venkata Rama Rao Bommireddipalli is in a meeting ::
:: grapek@us.ibm.com - Howard Grapek/Boulder/Contr/IBM is
available ::
available ::
chat. ::
:: sreekanthj@in.ibm.com - Sreekanth Jayaprakash/India/IBM is offline ::
:: Sheelnidhi Gupta is away ::
sent a quick status 09 PM
? 11:53:
i will work on adding some more details as soon as I can 23 PM
:: Sheelnidhi Gupta is available ::
:: Pedro Matheus Dias left the chat. ::
Frank D. (... 12:24:
marty where is the location you are snapping 49 AM
chat. ::
offline ::
:: jamartin@us.ibm.com - Jo-Ann Martin/Southbury/IBM left the chat. ::
:: Manish H Ghildyal left the chat. ::
:: Avinash Gautam is away ::
Marty Heb... 12:57:
http://www.microsoft.com/windowsserver2003/technologies/manage 35 AM
ment/powershell/download.mspx
:: Avinash Gautam is available ::
the chat. ::
available ::
:: zoharmann@us.ibm.com - Zohar Mann/Los Angeles/IBM is away ::
chat. ::
:: grapek@us.ibm.com - Howard Grapek/Boulder/Contr/IBM left the chat.
::
:: Avinash Gautam left the chat. ::
:: Sheelnidhi Gupta left the chat. ::
:: Sheelnidhi Gupta is offline ::
Frank D. (... 1:37:2
Here is the script 0 AM
? 1:37:3
I tested and it works fine. 2 AM
Frank D. (... 1:37:3
robocopy c:\test1 c:\test /MIR /X0 /COPY:DATSOU /R:1 /W:1 /TEE 4 AM
/V /NP /LOG+:test
? 1:37:5
we will change source and dest. based on needs.. 0 AM
ok 6 AM
:: Avinash Gautam joined the chat. ::
meeting ::
the chat. ::
available ::
chat. ::
zoharman... 2:17:1
Hi Marty. You guys getting ready for turnover? 6 AM
Marty Heb... 2:17:2
should be soon. we said 5 pm est 8 AM
zoharman... 2:17:5
I'd like to be on that call pls 0 AM
Marty Heb... 2:18:1
OK I will tell you when I hear from Daniel 1 AM
:: Sheelnidhi Gupta left the chat. ::
Frank D. (... 2:19:5
marty 1 AM
Marty Heb... 2:19:5
yes 6 AM
Frank D. (... 2:20:0
snap is going to what filer 3 AM
? 2:20:1
and a share called temp 8 AM
Marty Heb... 2:20:1
same erdhfdnas002 9 AM
:: Avinash Gautam is away ::
Frank D. (... 2:20:2
so \\erdhfdnas002\temp 8 AM
Marty Heb... 2:20:3
the share does not exist until the snap is done, but yes 4 AM
:: Avinash Gautam is available ::
:: Avinash Gautam left the chat. ::
:: qmohamme@in.ibm.com - Qamaruddin A Mohammed/India/IBM
joined the chat. ::
Kishore Gu... 2:49:0
Qamar from our team will be joining...I am leaving for the day 3 AM
qmohamm... 2:50:0
Hi All 0 AM
hello 9 AM
:: Daniel Esposito *Contractor* joined the chat. ::
:: Kishore Guduru left the chat. ::
Daniel Esp... 2:53:1
Good evening, taking over NAS responsibility from Marty, performing a 1 AM
turnover now
Frank D. (... 2:53:2
Hey dan 6 AM
Marty Heb... 2:53:2
is the call still going on? 8 AM
Frank D. (... 2:53:3
yes 1 AM
? 2:53:3
call is on. 8 AM
? 2:53:5
Dan when copy is done. 4 AM
? 2:54:0
Can we get a report on Qtree Size.. 8 AM
one sec 7 AM
Frank D. (... 2:54:2
to see about how much data is missing. 0 AM
1.1tb copied so far 6 AM
Frank D. (... 2:55:0
marty before you leave.. when was this kicked off. 8 AM
Marty Heb... 2:55:3
9:41 AM EST 4 AM
? 2:55:4
we are at 1121 9 AM
? 2:55:5
gb 1 AM
Frank D. (... 2:55:5
so we are 50 percent 5 AM
Marty Heb... 2:56:1
yes 5 AM
Frank D. (... 2:56:5
so we might as well break from call as we will not be able to start till 9 AM
10 or 11 easy
the chat. ::
available ::
zoharman... 3:03:2
Belinda- you there? 7 AM
yes 1 AM
zoharman... 3:04:3
I'm getting iinquiries for status updates from upper mgmt. Are you 7 AM
sending any more of those out?
zoharman... 3:06:4
Regarding the theory of this being possibly caused by a botched 2 AM
restore- have we seen any detailed logs from the backup infrastructure
or NAS filer supporting this idea?
we are going to close the bridge line...it will reopen at 6:30 pm ET for 1 AM
the next status
zoharman... 3:07:4
Also, Daniel will research turning on CIFS auditing with a high enough 3 AM
level to capture user IDs and actions, if needed.
is there anyone from netbackup on the chat? 7 AM
Shivaram ... 3:11:1
Hi daniel, i am from netbackup 0 AM
Do you know anything about this backup job that hung yesterday? 4 AM
Shivaram ... 3:15:2
can you please give me details of backup job 9 AM
:: Gerald L. Gase is away ::
from was was passed onto me, yesterday afternoon, somewhere 8 AM
around 1300est, a restore job was kicked off for this data location
? 3:23:1
that restore ran for a certain amount of time, appeared to be hung and 0 AM
had to be killed
morinmg@... 3:23:4
belinda 2 AM
? 3:24:1
can i get a quick update on this issue 8 AM
Shivaram ... 3:24:2
yes the restore was hung, it was not kileed but it failed 4 AM
can you send me the job report? 8 AM
? 3:24:4
verbatim? 6 AM
? 3:24:5
(verbose) 4 AM
sure, Mike... 7 AM
? 3:27:2
the copy of data is just over halfway complete 3 AM
? 3:27:5
we are running scripts to remove all the 0 kb files now... there are 1 AM
some issues with permissions but Jim from security is working thru
those
morinmg@... 3:28:1
if it's easier to talk i can call 1 AM
sure 6 AM
? 3:28:2
912-332-7702 0 AM
Shivaram, can you get the job log for the restore? 6 AM
Shivaram ... 3:30:2
just give me 2 min, checking 0 AM
8 AM
thanks
Frank D. (... 3:45:0
Dan where are we on copy. 1 AM
one sec, will check 9 AM
1.26tb 8 AM
available ::
:: zoharmann@us.ibm.com - Zohar Mann/Los Angeles/IBM left the
chat. ::
Using NetBackup (Version 5.0) to restore files from a Network 7 AM
Appliance (Using NDMP) restores files with correct names to the
correct location but the files have no data. Out of 23 files, only 3
were restored correctly with the data intact. All other files fromt he
restore were restored with 0 Kb of data in them and of course were
not
able to be opened. We have tried to turn off DAR on the server and
tried to restore more files after a manual backup without any luck.
bridge is open 6 AM
? 4:01:5
Shivaram, can you call syemantec and open a case asking about this 3 AM
in the current cersion
? 4:02:2
*version 4 AM
available ::
Shivaram ... 4:05:4
ok, i am logging the case] 5 AM
thank you 8 AM
available ::
available ::
chat. ::
qmohamm... 4:43:5
I am leaving for the day Narsima Peddi is joining the chat 1 AM
:: Narsimhulu Peddi joined the chat. ::
Narsimhul... 4:45:4
Hi All 6 AM
:: qmohamme@in.ibm.com - Qamaruddin A Mohammed/India/IBM left
the chat. ::
:: Frank D. (Frank) Breen is away ::
welcome 3 AM
? 4:53:4
we are just monitoring the data copy and working with netbackup 4 AM
vendor at this time
Narsimhul... 4:53:5
ok 9 AM
if you can just monitor the chat room in case you are needed, that'd 1 AM
be great
Narsimhul... 4:54:1
ok 6 AM
i'm going to step away for awhile... but will check status of the chat 3 AM
room every now and again... the next call is at 10pm ET
:: Belinda Brotherson *Contractor* is away ::
:: Narsimhulu Peddi is offline ::
:: John Fritz *Contractor* is offline ::
:: Gerald L. Gase is offline ::
:: Shivaram Aswathaiah is offline ::
:: Ritu Sharma is offline ::
:: Belinda Brotherson *Contractor* is offline ::
:: Frank D. (Frank) Breen is offline ::
:: Daniel Esposito *Contractor* is offline ::
:: Bhavani S Ande joined the chat. ::
the chat. ::
chat. ::
sorry guys...i'm still stuck in morning status calls...will be back as soon 4 PM
as i can
Frank D. (... 5:39:4
sure np 7 PM
:: vinamboo@in.ibm.com - Vishal Namboori1/India/IBM is away ::
155gb 1 PM
158gb 3 PM
? 6:25:2
biggest problem with the speed at which we repopulate data is the 0 PM
amount of data that has to be scanned through
Frank D. (... 6:25:4
yes 2 PM
28% 2615 5857 0 13237 15431 17045 24318 0 0 30 1 PM
31% 2349 5804 0 19669 23476 17478 24646 0 0 30
53% 1767 5676 0 47792 52986 17770 16946 0 0 31
49% 2056 5893 0 23131 27580 22635 12 0 0 31
45% 2121 4322 0 13995 14993 24120 16 0 0 31
45% 1896 3904 0 17894 19635 22603 0 0 0 31
53% 1725 4147 0 50762 52072 14404 12 0 0 31
available ::
available ::
shaswath@in.ibm.com - Shivaram Aswathaiah/India/IBM: The restore 7 PM
command writes file system data from tape to disk using a predefined
process.
The process in the following table shows how the restore command
works.
Stage Action
1 Data ONTAP catalogs the files that need to be extracted from the
tape.
2 Data ONTAP creates directories and empty files.
3. Data ONTAP reads a file from tape, writes it to disk, and sets the
permissions (including ACLs) on it.
? 6:55:3
http://now.netapp.com/NOW/knowledge/docs/ontap/rel657_gf/html/o 3 PM
ntap/tapebkup/recover4.htm\
gotta run to another meeting... bridge will stay open... and i will be 0 PM
back soon
Marty Heb... 7:06:4
Filesystem total used avail capacity Mounted on 2 PM
:: Manish K Srivastava joined the chat. ::

:: Manish K Srivastava left the chat. ::
back 9 PM
welcome 7 PM
168gb 6 PM
:: Bhavani S Ande left the chat. ::
:: Bhavani S Ande is offline ::
Marty Heb... 7:18:4
We have 629 users are attached to the CMSshrdata share. 4 PM
John Fritz ... 7:23:3
07/30/2010 9 PM
Frank D. (... 7:27:3
dan i fired up 2 more servers. 3 PM
? 7:27:4
check NAS status. 1 PM
Marty Heb... 7:28:1
45 of the 629 are write locks the rest are read only locks. 1 PM
:: Gerald L. Gase is away ::
:: Gerald L. Gase is available ::
is the bridge still ok? 2 PM
Marty Heb... 7:33:4
yes 2 PM
ok, i had to drop from the call per my crazy other phone line 9 PM
Marty Heb... 7:34:2
169GB 1 PM
7.8% complete 9 PM
Frank D. (... 7:36:5
how is nas. perf 8 PM
? 7:37:0
7 systems 3 PM
? 7:37:0
now 5 PM
Marty Heb... 7:50:5
172GB 5 PM
whats the magic number we need to get to 2.2? 8 PM
Frank D. (... 7:51:3
2002 2 PM
? 7:51:3
2200 7 PM
Marty Heb... 7:52:0
CPU NFS CIFS HTTP Net kB/s Disk kB/s Tape kB/s Cache 1 PM
in out read write read write age
20% 51 4683 0 22732 27526 17139 17818 0 0 19
86% 60 3102 0 14972 20542 21446 2691 0 0 19
49% 59 5754 0 18866 24994 10764 16064 0 0 19
22% 91 3601 0 17350 22071 15656 18458 0 0 19
31% 68 3890 0 16922 22520 5333 14515 0 0 19
27% 89 5680 0 25943 39285 24259 15988 0 0 19
96% 65 5541 0 11711 22454 30738 11 0 0 20
96% 65 2497 0 7932 14059 13550 4354 0 0 20
27% 143 8285 0 25575 39915 14933 24915 0 0 20
34% 82 8386 0 19119 19889 13200 14150 0 0 20
23% 46 5547 0 10743 16233 11141 12663 0 0 20
35% 42 4817 0 9628 15585 16960 11092 0 0 20
56% 59 5753 0 11286 16446 26204 24 0 0 20
89% 61 2424 0 6102 9168 5724 2697 0 0 20
20% 84 5945 0 13122 19582 11976 15636 0 0 20
20% 49 3942 0 10595 16364 8623 11788 0 0 20
24% 47 3462 0 11701 17325 14867 9763 0 0 20
27% 32 2557 0 13653 19907 17201 0 0 0 20
30% 50 3881 0 10948 16775 18265 24 0 0 20
:: Gerald L. Gase left the chat. ::

Frank D. (... 8:06:1
860-281-5552 - cell 2 PM
\\erdhfdnas002\CMSShrData\Retention 4 PM
? 8:14:4
\\erdhfdnas002\CMSShrData\Filings Assistance 1 PM
\\erdhfdnas002\CMSShrData\Spectrum Product Development

Marty Heb... 8:31:1
179GB 7 PM
:: riyaz.ahmed@in.ibm.com - Riyaz A Mela/India/IBM joined the chat. ::
:: riyaz.ahmed@in.ibm.com - Riyaz A Mela/India/IBM is available ::
thank you 8 PM
:: riyaz.ahmed@in.ibm.com - Riyaz A Mela/India/IBM is away ::
Marty Heb... 8:39:1
bash-2.05$ rsh erdhfdnas002 df -V -g HFD_flx_306 9 PM
:: riyaz.ahmed@in.ibm.com - Riyaz A Mela/India/IBM is available ::

Marty Heb... 8:40:0
CPU NFS CIFS HTTP Net kB/s Disk kB/s Tape kB/s Cache 0 PM
27% 44 5127 0 43598 40341 16408 26805 0 0 34
24% 68 4999 0 40761 37075 13789 23203 0 0 34
20% 89 3969 0 26455 23765 12084 26621 0 0 34
27% 57 4678 0 32844 24111 14062 23057 0 0 32
71% 69 2866 0 28209 20756 21844 45792 0 0 32
64% 55 3464 0 29803 26074 24884 25801 0 0 32
37% 39 4587 0 29807 27530 14080 24232 0 0 32
26% 30 4303 0 28128 26225 20427 25934 0 0 32
25% 32 4014 0 39090 35037 17147 22725 0 0 32
28% 45 4750 0 56342 51980 17390 25331 0 0 32
42% 61 5155 0 64761 56804 14809 38625 0 0 33
30% 51 4429 0 54381 50960 30339 48454 0 0 33
32% 66 4890 0 58097 52951 19972 41126 0 0 33
62% 59 4081 0 40055 37216 27503 55074 0 0 33
86% 32 2396 0 21844 25517 15828 21642 0 0 33
29% 58 4920 0 46919 54310 37825 27223 0 0 33
25% 42 3880 0 47323 48393 21934 23568 0 0 33
31% 37 4071 0 45447 46328 18865 24000 0 0 35
34% 64 4168 0 45100 44620 24181 44919 0 0 35

:: Gerald L. Gase is available ::
chat. ::
:: zoharmann@us.ibm.com - Zohar Mann/Los Angeles/IBM is available ::
Frank D. (... 8:45:2
John these are done. 8 PM
? 8:45:2
Spectrum Product Development 9 PM
? 8:45:4
RetentionScores 9 PM
? 8:46:3
Retention Analysis 1 PM
Frank D. (... 8:47:0
The rest are still going. 1 PM
Frank D. (... 8:54:2
Retention Dividends Corp File 1 PM
? 8:54:2
done 3 PM
Marty Heb... 8:57:0
:: riyaz.ahmed@in.ibm.com - Riyaz A Mela/India/IBM left the chat. ::

:: riyaz.ahmed@in.ibm.com - Riyaz A Mela/India/IBM is offline ::
Marty Heb... 9:07:2
bash-2.05$ rsh erdhfdnas002 sysstat 1 9 PM
CPU NFS CIFS HTTP Net kB/s Disk kB/s Tape kB/s Cache
36% 55 5101 0 56768 59196 21504 31100 0 0 22
76% 31 2585 0 33680 34522 27165 32035 0 0 22
91% 132 4994 0 36445 40723 31869 83877 0 0 22
41% 636 4453 0 52819 56994 29126 46419 0 0 19
37% 1238 4938 0 56631 59139 22766 50109 0 0 19
47% 756 4326 0 54182 56634 18344 50096 0 0 19
35% 6 4501 0 50299 54663 20971 48615 0 0 19
37% 15 5349 0 58083 62332 19265 51001 0 0 19
73% 11 2829 0 28798 31963 18426 38067 0 0 22
97% 21 3570 0 33610 37996 20518 61109 0 0 22
43% 28 6576 0 63395 69606 30885 46845 0 0 22
35% 18 5187 0 54606 62878 19785 51350 0 0 22
37% 33 5088 0 57747 61590 20809 46406 0 0 22
40% 75 5053 0 53450 56622 24862 49541 0 0 22
76% 181 3235 0 30789 33084 20577 35005 0 0 22
58% 454 6580 0 60337 60595 29024 65821 0 0 22
42% 464 5456 0 60746 64478 35471 41678 0 0 22
41% 301 5564 0 63201 64601 24415 53184 0 0 22
46% 250 4867 0 53044 55195 29904 50028 0 0 20
42% 191 5123 0 53786 61460 26604 40414 0 0 20
^C 93% 14 3533 0 26060 29710 17292 39033 0 0 21
bash-2.05$ rsh erdhfdnas001 sysstat 1
23% 1816 3123 0 1952 8080 7353 24 0 0 58
23% 1901 3051 0 2674 8580 12191 0 0 0 58
26% 1933 3181 0 5313 8243 10245 7649 0 0 58
17% 1560 3183 0 4196 8485 13998 26401 0 0 58
31% 1894 3355 0 7234 9047 12496 14050 0 0 58
23% 1512 3225 0 2572 8986 11649 0 0 0 58
29% 217 2987 0 3833 9901 8232 24 0 0 57
24% 368 2702 0 3995 10420 11405 0 0 0 58
19% 177 3495 0 2626 9907 8675 8 0 0 58
27% 180 3041 0 3756 9886 10123 23 0 0 58
21% 197 3957 0 4210 10562 11505 0 0 0 58
21% 202 2419 0 1759 11258 11677 0 0 0 58
33% 809 2865 0 3136 11584 12291 24 0 0 58
22% 81 2753 0 1453 10987 14399 8 0 0 58
33% 497 2937 0 2533 12344 18867 7840 0 0 58
27% 2408 2980 0 2621 12622 9646 27437 0 0 58
39% 1727 3042 0 2042 12516 16294 17306 0 0 58
33% 973 2877 0 16781 11974 11366 0 0 0 58
23% 480 3360 0 1926 13457 12032 665 0 0 58
24% 308 3108 0 2332 12940 13337 0 0 0 58
23% 392 3910 0 2543 14121 9236 0 0 0 58

available ::
available ::
Marty Heb... 9:29:4
209GB 6 PM
meeting ::
chat. ::
Marty Heb... 9:34:5
02:a0:98:08:95:0a 9 PM
? 9:35:3
157.209.175.202 9 PM
? 9:44:0

Frank D. (... 9:56:2
\\Erdhfdnas002\Cmsshrdata\00-P&CTD\HSI\Training Coordinator 1 PM
Files\letter
? 9:56:2
done 2 PM
Marty Heb... 9:58:5
Frank D. (... 10:03:

Filings Assistance 55 PM
:: Daniel Esposito *Contractor* is away ::
Marty Heb... 10:10:
972-533-2716 45 PM
? 10:16:
:: zoharmann@us.ibm.com - Zohar Mann/Los Angeles/IBM is available ::

:: Shivaram Aswathaiah joined the chat. ::
chat. ::
Marty Heb... 10:57:

John Fritz ... 10:58:
thank you 45 PM
? 10:59:
BTW Marty Howie maybe ready for some EV resolution activity in 21 PM
about 15 minutes, we will see.
? 10:59:
BRB 10-15 I have my phones if needed sooner 48 PM
chat. ::
available ::
:: Belinda Brotherson *Contractor* left the chat. ::
Marty Heb... 11:17:
:: Frank D. (Frank) Breen is available ::

Frank D. (... 11:33:
John.. did you get this is done. 41 PM
? 11:33:
42 PM
Filings Assistance
? 11:34:
Spectrum Product Development is done 54 PM
? 11:36:
erdhfdnas002\HFD_flx_306$\CMSdata\CMSdata\ShrData\Controllers is 16 PM
done
not yet Frank i'll pass thoise on to Joe B now that : 30 PM
? 11:36:
Filings Assistance 54 PM
Spectrum Product Development
erdhfdnas002\HFD_flx_306$\CMSdata\CMSdata\ShrData\Controllers is
done
Frank D. (... 11:36:
\\erdhfdnas002\HFD_flx_306$\CMSdata\CMSdata\ShrData\BI Staffing 57 PM
is done
are all done 58 PM
Frank D. (... 11:37:
yes 01 PM
? 11:37:
that I listed here. 15 PM
? 11:37:
not all on xls 30 PM
? 11:37:
so the 4 i said are done,. 56 PM
? 11:38:
ilings Assistance 06 PM
Spectrum Product Development
erdhfdnas002\HFD_flx_306$\CMSdata\CMSdata\ShrData\Controllers is
done
2:06:56 PM: fbreen@us.ibm.com - Frank Breen/Hartford/IBM:
\\erdhfdnas002\HFD_flx_306$\CMSdata\CMSdata\ShrData\BI Staffing
is done
OK, Frank, I got 4 done now to pass along that Joe B does not have 14 PM
yet, thank you.
Frank D. (... 11:42:
k 13 PM
? 11:45:
Are you on the bridge. 04 PM
no we were going to open it at 3pm again 41 PM
Frank D. (... 11:45:
ok 44 PM
? 11:45:
finr 46 PM
Do we need to talk? 47 PM
Frank D. (... 11:45:
can wait till 3 59 PM
I was hopin you were sleeping 07 PM
Frank D. (... 11:46:
setting up more jobs 36 PM
great! 40 PM
? 11:47:
Marty? 35 PM
Marty Heb... 11:47:
yes 46 PM
? 11:47:
282 gb 58 PM
? 11:48:

thank ou 44 PM
? 11:48:
you 47 PM
Marty Heb... 11:48:
np 54 PM
? 12:13:
All most at 300 55 AM


chat. ::
Kelly K. Cota 12:19:
what time is the next yellow event update John 41 AM
6pm 45 AM
can you please ensure Sheri Beckett is on copy , she is covering for 10 AM
Nichole Williford, thanks
I can try. 15 AM
ty 51 AM
ymw 59 AM
Frank D. (... 12:34:
john 23 AM
? 12:34:
bridge # 28 AM
Marty Heb... 12:36:
Filesystem total used avail capacity Mounted on 08 AM
Frank D. (... 12:36:

marty. 37 AM
Marty Heb... 12:36:
Frank 42 AM
Frank D. (... 12:36:
can you give me the # 46 AM
Marty Heb... 12:36:
320 GB 52 AM
Frank D. (... 12:36:
Bridge 56 AM
? 12:36:
line 58 AM
Marty Heb... 12:36:
oooh 59 AM
? 12:37:
4210624480
chat. ::
meeting ::
Marty can you check and post a new status complete 0 AM
? 1:31:0
? 1 AM
Marty Heb... 1:31:2

super 4 AM
? 1:32:0
thank you 4 AM
Marty Heb... 1:37:3
current performance 6 AM
bash-2.05$ rsh erdhfdnas002 sysstat 1

33% 11 5025 0 26187 34057 34466 39161 0 0 15
43% 4 4833 0 38336 49259 24655 69750 0 0 15
34% 1 4816 0 40500 48287 28175 25143 0 0 15
38% 13 5166 0 43733 50925 16056 51682 0 0 15
37% 79 4808 0 36685 43399 23892 42234 0 0 15
93% 26 2004 0 8708 12349 38130 19762 0 0 15
99% 91 4556 0 22143 27206 45451 69369 0 0 15
73% 32 3276 0 24965 30884 28126 20279 0 0 15
73% 178 7697 0 45791 53701 63201 51877 0 0 15
69% 121 3715 0 20764 26644 43113 42867 0 0 15
54% 18 5082 0 34653 48149 40064 25968 0 0 15
87% 20 2291 0 17007 26883 30184 18193 0 0 15
56% 40 4306 0 41790 58487 48771 19416 0 0 14
39% 24 4888 0 47242 57620 39956 22299 0 0 14
57% 20 4022 0 39718 52741 33314 17326 0 0 14
41% 11 5236 0 57500 65332 29799 20949 0 0 14
? 1:58:2
? 2:03:5
Hey John, what time are we going to meet again on the brdige? 7 AM
hm 8 AM
? 2:05:1
I hat eot say I think we need a short call at 5 1 AM
Marty Heb... 2:05:2
ok 5 est 1 AM
Frank D. (... 2:05:2
can we do it now. 2 AM
Marty Heb... 2:05:2
sure 6 AM
Frank D. (... 2:05:3
as I am going to head out to see whats up 9 AM
sure 6 AM
Marty Heb... 2:07:3

chat. ::
:: zoharmann@us.ibm.com - Zohar Mann/Los Angeles/IBM is offline ::
Marty Heb... 2:18:3
:: Daniel Esposito *Contractor* is available ::

:: Kelly K. Cota is away ::
Marty Heb... 2:28:3
Diali in check in 6 AM
9:00 PM
8:00 AM
10:30 AM
4:30 PM
? 2:28:5
4210624480
? 2:33:5
My mobile 972-533-2716 9 AM
Frank's 860-281-5552 0 AM
? 2:44:5
Marty, can you post a % curently 5 AM
Marty Heb... 2:46:2

Marty can you tell when files are restored? 9 AM
Marty Heb... 2:47:0
That I can not see. 0 AM
that's wht I thought 7 AM
Marty Heb... 2:47:2
Need frank or one of his team 2 AM
yes 8 AM
:: davidal@us.ibm.com - David Allen/Boulder/IBM joined the chat. ::
Frank D. (... 2:54:2
whats up 1 AM
? 2:54:2
what do you need. 6 AM
good afternoon 0 AM
chat. ::
Frank D. (... 2:55:4
i guess you could call it that. 8 AM
meeting ::
lol, how is everything going Frank? 7 AM
Frank D. (... 2:56:4
Dest = 9 AM
\\erdhfdnas002\HFD_flx_306$\CMSdata\CMSdata\ShrData\WCLOB\ is
retored
? 2:56:5
not good. 4 AM
? 2:56:5
not good. 6 AM
? 2:57:0
could be better. 5 AM
don't you have anyone to take this over for you? 5 AM
Frank D. (... 2:58:0
no. you only get me. 4 AM
you've been at it a while now, haven't you? 5 AM
Frank D. (... 3:02:3
yes 3 AM
? 3:02:4
its all good 5 AM
Marty Heb... 3:02:4
Frank go see your wife 7 AM
Frank D. (... 3:03:0
I got a call. 0 AM
? 3:03:2
she is alright and will come home. 2 AM
Marty Heb... 3:03:2
Good to know 9 AM
Frank D. (... 3:03:3
yes 2 AM
Gerald L. G... 3:03:3
Good news!! 2 AM
Marty Heb... 3:03:3
still wife comes before work 6 AM
Frank D. (... 3:04:1
So I am checking all the requested Jobs 4 AM
? 3:04:3
Gerald do you or Dave need this info 7 AM
super news on the wfe ! 3 AM
Frank D. (... 3:05:1
yeah. 4 AM
? 3:05:2
ty 3 AM
I need to know if we have started or scirpted. 6 AM
Frank D. (... 3:06:3
you have a new list. 1 AM
Marty Heb... 3:06:5
All, Daniel is taking over for me for a while. I am going to eat dinner 7 AM
Frank D. (... 3:08:4
marty enjoy.. 0 AM
gl marty 2 AM
not any more than wha tI fwd to you 9 AM
Frank D. (... 3:10:1
ok.. 5 AM
? 3:10:2
lets start with the latest two 8 AM
? 3:10:4
\\erdhfdnas002\CMSShrData\Billing is Runing 8 AM
? 3:11:0
\\erdhfdnas002\CMSShrData\WCLOB done 0 AM
chat. ::
chat. ::
meeting ::
Frank where do you want the update sent ? 5 AM
Frank D. (... 3:28:5
what update.. new things to restore. 5 AM
? 3:29:0
IBM 0 AM
the spread sheet when we send status 5 AM
:: davidal@us.ibm.com - David Allen/Boulder/IBM left the chat. ::
Frank D. (... 3:34:2
Ok 1 AM
? 3:34:3
Completed 5 AM
\\erdhfdnas002\CMSShrData\Retention
\\erdhfdnas002\CMSShrData\RetentionAnalysis
\\erdhfdnas002\cmsshrdata\Alternative Risk
\\erdhfdnas002\cmsshrdata\Alternative Risk\
\erdhfdnas002\cmsshrdata\BI Staffing
\\erdhfdnas002\cmsshrdata\Billing\PrintKey
\\erdhfdnas002\CMSShrData\Filings Assistance
\\erdhfdnas002\cmsshrdata\clcs commission project \production
support\release to production
\\erdhfdnas002\cmsshrdata\Rose P\Ky Tax Procedures\ky tax info
Frank D. (... 3:34:4
Running 7 AM
\\erdhfdnas002\cmsshrdata\QATMML\Validation DB\Application\Middle
Market\Production
\\ERDHFDNAS002\CMSSHRDATA\bkf it support\vss work\
\\erdhfdnas002\ECM\CMSshrdata
\Erdhfdnas002\Cmsshrdata\CL Product Strategy
Frank D. (... 3:35:0
Pending - setting up. 9 AM
? 3:35:1
\\erdhfdnas002\CMSshrdata\eBTSMarketingCommunications\EBC\Brok 3 AM
er of Record for CL
\\erdhfdnas002\cmsshrdata\SNERO_SC\FLOW
\\erdhfdnas002\CMSShrData\Spectrum Product
\\erdhfdnas002\CMSshrdata\WC Quote Rescore\Working Files
\\erdhfdnas002\QATMML\MM Northeast and SE\Local MML Stuff incl.
\\erdhfdnass002\Filings Assistance\Filing Material Cmsshrdata
\\ernashfdnas002\\Claims\Shared\Shared\Claim_Account_Managemen
t\DCAR
chat. ::
Gerald L. G... 3:46:1
Frank, question for you. In the spreadsheet we are carrying 1 AM
\\erdhfdnas002\cmsshrdata\Billing as a "hot item". Your update for
completed is \\erdhfdnas002\cmsshrdata\Billing\PrintKey. Is there
more of Billing? I assume a lot more. Will you update us when the
Parent completes? Make sense?
Frank D. (... 3:48:2
Billing is running 1 AM
? 3:48:2
not completed. 7 AM
? 3:48:5
\ 7 AM
ernashfdnas002\\Claims\Shared\Shared\Claim_Account_Management\
DCAR
? 3:49:0
can not find tis 2 AM
? 3:49:1
can not find this. 1 AM
? 3:49:1
I need a correct path. 6 AM
? 3:49:4
these are all scheduled and running. 1 AM
? 3:50:0
\erdhfdnas002\cmsshrdata\QATMML\Validation DB\Application\Middle 0 AM
Market\Production
\\erdhfdnas002\QATMML\MM Northeast and SE\Local MML Stuff incl.
\\ERDHFDNAS002\CMSSHRDATA\bkf it support\vss work\
\Erdhfdnas002\Cmsshrdata\CL Product Strategy
\\erdhfdnas002\CMSshrdata\eBTSMarketingCommunications\EBC\Brok
er of Record for CL
\\erdhfdnas002\CMSshrdata\WC Quote Rescore\Working Files
? 3:52:2
\billing\CCS\Meetings\2005\ 0 AM
:: John Fritz *Contractor* is away ::
Frank D. (... 3:52:3
current location of Billing restore. 4 AM
Gerald L. G... 3:54:4
Frank, we are going to send you the update communication with the 7 AM
updated spreadsheet. Which side do you want it? Outlook or Notes.
Frank D. (... 3:55:3
Dest = \\erdhfdnas002\HFD_flx_306$\CMSdata\CMSdata\ShrData\WC 6 AM
Quote Rescore\
? 3:55:4
completed. 0 AM
? 3:56:0
notes 3 AM
Gerald L. G... 3:57:3
OK. Looks like it already went to your Outlook side but I am 5 AM
forwarding to Notes.
Frank D. (... 3:59:2
billing is 200gb 2 AM
? 3:59:3
we have restored 89gb 1 AM
? 3:59:3
so far 4 AM
Gerald L. G... 3:59:4
OK. Thanks. 1 AM
? 4:02:1
I think I know this answer so bear with me for the dumb 5 AM
question...
\\erdhfdnas002\HFD_flx_306$\CMSdata\CMSdata\ShrData\WC Quote
Rescore\ is complete. Does that include the sub directory Working
Files? (\\erdhfdnas002\HFD_flx_306$\CMSdata\CMSdata\ShrData\WC
Quote Rescore\Working Files)
Frank D. (... 4:02:3
yes 5 AM
508gb 8 AM
Gerald L. G... 4:03:4
Thanks and sorry. I figured that was the way the restore worked but 0 AM
just needed to make sure before marking "Working Directory" off.
? 4:07:4
And Dan, Marty was pasting the following. 6 AM
So with your update are you saying....

Frank D. (... 4:11:2
Dest = \\erdhfdnas002\HFD_flx_306$\CMSdata\CMSdata\ShrData\CL 7 AM
Product Strategy\
? 4:11:3
Completed 0 AM
? 4:11:5
Dest = 3 AM
\\erdhfdnas002\HFD_flx_306$\CMSdata\CMSdata\ShrData\Rose P\
? 4:11:5
Completed 6 AM
1.1gb/minute, 25 hours remaining at that rate 6 AM
Frank D. (... 4:12:3
Dest = \\erdhfdnas002\HFD_flx_306$\CMSdata\CMSdata\ShrData\BI 0 AM
Staffing\
? 4:12:3
Completed 5 AM
:: Kelly K. Cota is offline ::
:: Shivaram Aswathaiah is offline ::
:: madhusudan.kapai@in.ibm.com - Madhusudan Kapai/India/IBM joined
the chat. ::
chat. ::
Frank D. (... 7:15:2
ok 7 PM
? 7:46:4
Dan or Marty.. numbers should be up. 6 PM
? 7:47:1
I have each server running multiple scripts against very large restore 1 PM
points.
:: madhusudan.kapai@in.ibm.com - Madhusudan Kapai/India/IBM is
available ::
:: davidal@us.ibm.com - David Allen/Boulder/IBM is available ::
wow, 873gb 7 PM
? 8:02:0
60% 28 5516 0 31446 32604 33375 38724 0 0 19 4 PM
44% 55 7770 0 42246 44172 44451 57007 0 0 19
61% 314 7497 0 35550 38067 42304 47798 0 0 19
51% 358 9182 0 38774 41532 40535 47293 0 0 19
77% 43 7569 0 32725 35794 46703 38839 0 0 19
64% 72 7986 0 33966 37366 41838 47647 0 0 19
60% 58 6803 0 27580 30697 35446 26481 0 0 19
95% 60 5820 0 28692 29813 39817 30386 0 0 19
53% 67 9800 0 39496 40258 40933 40919 0 0 19
50% 102 9383 0 40284 41930 40486 45449 0 0 16
53% 65 9357 0 43400 45915 45558 43930 0 0 16
78% 55 7344 0 34545 36415 43166 11905 0 0 16
Frank D. (... 8:02:1
yeah. 5 PM
? 8:02:2
seems better 3 PM
7/31/2010 6:00 747 72 245 0.29 5 PM
7/31/2010 7:18 764 17 78 0.22
7/31/2010 7:55 775 11 37 0.30
7/31/2010 10:31 873 98 156 0.63
Average gb/min 0.48
1/0/1900 2:36
? 8:04:5
it was 2 hours 36 minutes since the last status 7 PM
? 8:05:2
41% 8 PM
? 8:08:1
this is where Frank should take credit 9 PM
? 8:17:2
7/31/2010 10:31 873 98 156 0.63 0 PM
7/31/2010 10:46 899 26 15 1.73
Average gb/min 0.52

:: davidal@us.ibm.com - David Allen/Boulder/IBM is offline ::
offline ::
chat. ::
:: madhusudan.kapai@in.ibm.com - Madhusudan Kapai/India/IBM joined
the chat. ::
offline ::
chat. ::
:: Manish K Srivastava is offline ::
chat. ::
chat. ::
:: andrestp@us.ibm.com - Andre St Pierre/Phoenix/Contr/IBM joined the
chat. ::
:: andrestp@us.ibm.com - Andre St Pierre/Phoenix/Contr/IBM is available
::
available ::
davidal@u... 6:20:0
Daniel, can I get the latest update, please? 2 PM
Date / Time Total Increase Since Last Update Minutes Since Last 4 PM
Update Rate / Minute
8/1/2010 1:50 1827

8/1/2010 8:43 1914 87 413 0.21
Average gb/min 2.02
Percent Complete 0.90
davidal@u... 6:20:1
1 PM
doh.. read my mind
3 PM
sure
:: sreekanthj@in.ibm.com - Sreekanth Jayaprakash/India/IBM is away ::
available ::
8/1/2010 1:50 1827

8/1/2010 8:43 1914 87 413 0.21
8/1/2010 8:58 1935 21 15 1.40
Average gb/min 2.02

? 6:31:2
speed is picking up again 4 PM
davidal@u... 6:31:4
Maybe Frank kicked off some more jobs 1 PM
? 6:31:4
? 3 PM
54% 757 3232 0 29298 38571 106451 44399 0 47283 10 7 PM
58% 110 4289 0 32516 42912 97634 40787 0 50019 10
42% 23 3435 0 33175 43302 106914 100944 0 51932 10
43% 105 3373 0 29993 38513 96271 80161 0 52846 10
55% 76 3575 0 29339 38814 112836 21359 0 65373 10
? 6:32:2
that's what I am thinking 0 PM
davidal@u... 6:34:0
yep, writes are back up 8 PM
Frank D. (... 6:44:5
yeah 7 PM
? 6:45:0
a few 0 PM
8/1/2010 1:50 1827
8/1/2010 8:43 1914 87 413 0.21
8/1/2010 8:58 1935 21 15 1.40
8/1/2010 9:23 1976 41 25 1.64
Average gb/min 1.74
? 6:55:2
brb 5 PM
Frank D. (... 7:06:0
David are you there. 2 PM
davidal@u... 7:06:1
yes, Frank, I'm here 4 PM
:: andrestp@us.ibm.com - Andre St Pierre/Phoenix/Contr/IBM is offline ::
chat. ::
chat. ::
:: Manish K Srivastava is offline ::
chat. ::
8/1/2010 1:50 1827
8/1/2010 8:43 1914 87 413 0.21
8/1/2010 8:58 1935 21 15 1.40
8/1/2010 9:23 1976 41 25 1.64
8/1/2010 10:38 2017 41 75 0.55
Average gb/min 1.74


available ::
chat. ::
8/1/2010 1:50 1827
8/1/2010 8:43 1914 87 413 0.21
8/1/2010 8:58 1935 21 15 1.40
8/1/2010 9:23 1976 41 25 1.64
8/1/2010 10:38 2017 41 75 0.55
8/1/2010 11:12 2020 3 34 0.09
Average gb/min 1.74

Frank D. (... 8:58:1

We are on X:\N Burch 8 PM
? 8:58:1
X is mapped 9 PM
? 8:58:1
to gold copy 9 PM
? 8:58:1
so we are moving. 9 PM
? 8:58:1
on master 9 PM
available ::
available ::
:: andrestp@us.ibm.com - Andre St Pierre/Phoenix/Contr/IBM is away ::
Frank D. (... 9:03:5
Largest dir left for copy are less than 3gb 8 PM
? 9:04:0
2gb 0 PM
2036 6 PM
? 9:06:1
8/1/2010 1:50 1827
8/1/2010 8:43 1914 87 413 0.21
8/1/2010 8:58 1935 21 15 1.40
8/1/2010 9:23 1976 41 25 1.64
8/1/2010 10:38 2017 41 75 0.55
8/1/2010 11:12 2020 3 34 0.09
8/1/2010 11:35 2036 16 23 0.70
Average gb/min 1.74

Frank D. (... 9:09:0
free space is holding steady 4 PM
? 9:09:1
322 8 PM
? 9:09:1
gb 8 PM
if needed, use cell 2 PM
Frank D. (... 9:26:2
folder 1090 the o's 5 PM
available ::
chat. ::
chat. ::
available ::
chat. ::
chat. ::
chat. ::
chat. ::
available ::
Daniel Esp... 10:59:
8/1/2010 9:23 1976 41 25 1.64
8/1/2010 10:38 2017 41 75 0.55
8/1/2010 11:12 2020 3 34 0.09
8/1/2010 11:35 2036 16 23 0.70
8/1/2010 12:38 2063 27 63 0.43
8/1/2010 13:28 2067 4 50 0.08
Average gb/min 1.28
davidal@u... 11:01:
thanks, Daniel 00 PM
:: davidal@us.ibm.com - David Allen/Boulder/IBM is away ::
::
available ::
Frank D. (... 11:27:
folder 1210 Product Fairs 06 PM
Daniel Esp... 11:36:
8/1/2010 9:23 1976 41 25 1.64
8/1/2010 10:38 2017 41 75 0.55
8/1/2010 11:12 2020 3 34 0.09
8/1/2010 11:35 2036 16 23 0.70
8/1/2010 12:38 2063 27 63 0.43
8/1/2010 13:28 2067 4 50 0.08
8/1/2010 16:02 2069 2 34 0.06
Average gb/min 1.21
? 11:36:
236gb free 48 PM
Frank D. (... 12:21:
I see 287 free - 2.02 tb restored. we are in the Q's 06 AM
? 12:21:
Folder 1250 22 AM
chat. ::
chat. ::
chat. ::
chat. ::
Marty Heb... 6:54:5
Date / Time Total Increase Since Last Update Minutes Since Last 3 AM
07/31/2010 10:46 899 26 15 1.73
07/31/2010 11:02 932 33 16 2.06
07/31/2010 11:15 955 23 13 1.77
07/31/2010 13:30 1167 212 135 1.57
07/31/2010 14:56 1331 164 86 1.91
07/31/2010 15:41 1444 113 45 2.51
07/31/2010 16:20 1546 102 39 2.62
07/31/2010 17:03 1627 81 43 1.88
07/31/2010 17:21 1657 30 18 1.67
07/31/2010 18:28 1715 58 67 0.87
07/31/2010 19:28 1728 13 60 0.22
07/31/2010 21:30 1731 3 122 0.02
07/31/2010 22:50 1774 43 80 0.54
08/01/2010 0:39 1815 41 109 0.38
08/01/2010 1:50 1827 12 71 0.17
08/01/2010 16:33 2097 270 883 0.31
08/01/2010 17:14 2107 10 41 0.24
08/01/2010 18:16 2115 8 62 0.13
08/01/2010 19:34 2128 13 78 0.17
08/01/2010 21:23 2146 18 109 0.17
Average gb/min 1.05

? 6:54:5
216 GB free 9 AM
Frank D. (... 6:56:2
Job has completed. 5 AM
? 6:56:3
all files restored. 2 AM
Marty Heb... 6:56:3
SWEET 6 AM
4 AM
? 6:57:0
MArty, what's the latest numbers? I am not logged in 8 AM
Marty Heb... 6:57:2
I just pasted them. look up a little 5 AM
? 6:57:3
1 AM
oh, so those are final, cool 0 AM
Marty Heb... 6:57:4
bash-2.05$ rsh erdhfdnas002 df -V -g HFD_flx_306 7 AM
? 6:57:4
here is the df 9 AM
do we have a chat lined up for the morning? 8 AM
Marty Heb... 6:58:4
Don't know. probably a good idea 5 AM
davidal@u... 6:59:0
oh. good news!! 2 AM
? 6:59:1
I can start one about 7:30 if that works for folks 7 AM
Marty Heb... 6:59:2
Works for me 3 AM
davidal@u... 6:59:5
I know Frank will be out in the morning.. 8 AM
? 7:00:0
Frank, should I invite Mike V to that? 5 AM
Frank D. (... 7:00:1
He has not called me back. 9 AM
davidal@u... 7:00:3
the first customer touchpoint is scheduled for 09:00, but I'm sure we'll 6 AM
give an update on the DOM call at 7;30, also
available ::
Frank D. (... 7:06:0
David .. Mike would not be able to go into any detail. 0 AM
? 7:06:1
So I would not suggest he attend. 2 AM
davidal@u... 7:06:5
okay.. I'll be giving status on the 7:30 0 AM
Frank D. (... 7:06:5
sure. 5 AM
davidal@u... 7:07:1
may need him or someone from your team on the 09:00 to determine 3 AM
the next steps
Frank D. (... 7:07:1
i will contact Mike in the AM. 4 AM
davidal@u... 7:07:5
k 6 AM
:: Frank D. (Frank) Breen left the chat. ::
::
available ::
:: andrestp@us.ibm.com - Andre St Pierre/Phoenix/Contr/IBM left the
chat. ::
chat. ::
::
:: Manish K Srivastava is away ::
:: Manish K Srivastava is available ::
:: Gerald L. Gase left the chat. ::

Chat

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Chat

Uploaded by

Copyright:

Available Formats

:: Kelly K. Cota joined the chat.

Fritz ie Moi 5:41:4

Kelly K. Cota 5:52:2

:: Hema Pabba joined the chat. ::

Belinda Br... 9:17:0

bash-2.05$ rsh erdhfdnas002 df -V -g HFD_flx_306

:: Manish K Srivastava joined the chat. ::

:: Gerald L. Gase left the chat. ::

\\erdhfdnas002\CMSShrData\Spectrum Product Development

:: riyaz.ahmed@in.ibm.com - Riyaz A Mela/India/IBM is available ::

:: Venkata Rama Rao Bommireddipalli is away ::

:: riyaz.ahmed@in.ibm.com - Riyaz A Mela/India/IBM left the chat. ::

:: morinmg@us.ibm.com - Michael G Morin/Poughkeepsie/IBM is

:: Venkata Rama Rao Bommireddipalli left the chat. ::

Frank D. (... 10:03:

:: zoharmann@us.ibm.com - Zohar Mann/Los Angeles/IBM is available ::

:: Ritu Sharma is away ::

:: Frank D. (Frank) Breen is available ::

John Fritz ... 11:48:

Filesystem total used avail capacity Mounted on

:: sreekanthj@in.ibm.com - Sreekanth Jayaprakash/India/IBM left the

Frank D. (... 12:36:

John Fritz ... 1:31:3

bash-2.05$ rsh erdhfdnas002 sysstat 1

:: zoharmann@us.ibm.com - Zohar Mann/Los Angeles/IBM left the

:: Daniel Esposito *Contractor* is available ::

John Fritz ... 2:46:4

So with your update are you saying....

Average gb/min 0.52

:: John Fritz *Contractor* is offline ::

8/1/2010 1:50 1827

8/1/2010 1:50 1827

Average gb/min 2.02

Average gb/min 1.74

:: sreekanthj@in.ibm.com - Sreekanth Jayaprakash/India/IBM is

Average gb/min 1.74

Frank D. (... 8:58:1

:: Ritu Sharma left the chat. ::

Average gb/min 1.05

You might also like

:: Daniel Esposito Contractor is available ::

:: John Fritz Contractor is offline ::