Webinar - Log/File Outputs

Processes - /var/log/dme/log
apic1# less dhcpd.bin.log
DHCP message transactions for APIC. DHCP allocation for infra tenant.

Look for obscured DN entry, here's why:

https://techzone.cisco.com/t5/Application-Centric/How-to-decode-log-file-DNs/ta-p/748025

982||16-02-27
11:23:21.881-08:00||firmware||DBG4||co=doer:255:127:0xff000000002636d1:1,d
n='"DlRMAAAEAAAAEAHBvZAAfAHQTAAABAAAAABAAAAAKAAAAAAAAAAAAAAAAAQIAAAA="'|

apic1# tail neo.log

Fuse mount points, file system cache, session management, REST access.

'912||139872550168320||2016-04-18 13:09:11,383||neo||INFO||Remote URL =

http://127.0.0.1:7777/api//mo/sys.xml?query-target=self||/mgmt/opt/control
ler/ishell/insieme/rest/restaccess.py||658'
'912||139872550168320||2016-04-18 13:09:11,387||neo||INFO||Remote URL =
http://127.0.0.1:7777/api//class/fabricNode.xml?||/mgmt/opt/controller/ish
ell/insieme/rest/restaccess.py||658'

apic1# tail nginx.bin.log

User, iNode, IFS authentications, image signing, RBAC/AAA/LDAP, security keys, licensing, authentication tokens

22619||16-04-18 14:33:02.089-07:00||aaa||DBG4||||Received response

from 10.29.181.22 - notifying callback handler
(IPv4)||../svc/extXMLApi/src/gen/ifc/app/./ping/lib_ifc_ping.cc||756 bico
00.615
22619||16-04-18
14:33:02.089-07:00||aaa||DBG4||||Received update on status of
10.29.181.22 (DN uni/userext/ldapext/ldapprovider-10.29.181.22) - status
is ALIVE||../svc/extXMLApi/src/gen/ifc/app/./pam/PamWorker.cc||1427

apic1# tail snoopy.log

Auto-completion for CLI commands.
 '543||2016-04-13 14:49:59,443||snoopy||INFO||TABCOMP: Completed args:
['Spine1', 'Leaf1', 'apic1', 'Leaf3',
'Leaf2']||/mgmt/opt/controller/ishell/insieme/cli/command_completer.py||16
8'

apic1# tail svc_ifc_appliancedirector.bin.log

IFC appliance cluster formation, heartbeat, cluster expansion, shrinking, sharding, sync of replica states.

998||16-04-18
14:34:36.430-07:00||replicator||INFO||co=repl:255:127:0xff00000000a56a26:6
||Creating
replication tx 0xff00000000a56a26 prev tx
0xff00000000a569fe||../controller/common/src/replicator/ReplicationTx.cc||
57

apic1# tail svc_ifc_dbgr.bin.log

Diagnostic of failure scenarios, logs, tech-support, SPAN, cores handling.

1984||16-04-18 14:35:53.816-07:00||dataexport||DBG4||||Successfully created

child; pid=31314||../common/src/dataexport/./ExportUtils.cc||391
1984||16-04-18 14:35:55.096-07:00||dataexport||DBG4||||Child exited with
status=0||../common/src/dataexport/./ExportUtils.cc||411 bico 53.816
1984||16-04-18
14:35:58.812-07:00||dataexport||DBG4||||running command
/mgmt/bin/leafPullJobDispatcher.py||../common/src/dataexport/./ExportUtils
.cc||349 bico 55.096
1984||16-04-18 14:35:58.817-07:00||dataexport||DBG4||||Successfully created
child; pid=31344||../common/src/dataexport/./ExportUtils.cc||391

apic1# tail svc_ifc_eventmgr.bin.log

Alerts/faults, events, health scores, syslog.

1383||16-04-18
14:37:32.864-07:00||event_||DBG4||co=doer:5:2:0x280000000053be26:1,dn='"D/
gcAAAcAVk13YXJlAA8AVDAxLUZvcldlYmluYXIADwBUMDEtRm9yV2ViaW5hcgA="',dn='"DQg
cAAAgAZGVmYXVsdAA="',fn=[doProcessFault]||EventProc(5)
Code=fault-F606351 ControlBits=4
Severity=4||../common/src/events/local/./FaultProcessor.cc||2226

apic1# tail svc_ifc_observer.bin.log

Collect and roll-up various hardware states.
 915||16-04-18
14:38:40.006-07:00||perf||DBG4||co=doer:255:127:0xff00000000346d73:1||*
Per Observable Performance Counters
*||../common/src/statAccess/./PerfCounters.cc||204

apic1# tail svc_ifc_policymgr.bin.log

Repository for all policies in the fabric, policy resolution, distribution of policies to embedded policy elements.

1560||16-04-18 14:33:07.505-07:00||net||DBG4||||Calling
tzset||../common/src/osiris/./core/utils/os_unix/Utils.cc||323 bico
07.488
1560||16-04-18 14:34:07.522-07:00||net||DBG4||||Calling
tzset||../common/src/osiris/./core/utils/os_unix/Utils.cc||323 bico
07.505
1563||16-04-18 14:35:07.538-07:00||net||DBG4||||Calling
tzset||../common/src/osiris/./core/utils/os_unix/Utils.cc||323 bico
07.522
1561||16-04-18 14:36:07.557-07:00||net||DBG4||||Calling
tzset||../common/src/osiris/./core/utils/os_unix/Utils.cc||323 bico
07.533

apic1# tail svc_ifc_reader.bin.log

Service API response data by reading and aggregating information directly from ObjectStore.

921||16-04-18 14:32:17.104-07:00||net||DBG4||||Calling
tzset||../common/src/osiris/./core/utils/os_unix/Utils.cc||323 bico
17.088
920||16-04-18 14:33:17.124-07:00||net||DBG4||||Calling
tzset||../common/src/osiris/./core/utils/os_unix/Utils.cc||323 bico
17.104
922||16-04-18 14:34:17.140-07:00||net||DBG4||||Calling
tzset||../common/src/osiris/./core/utils/os_unix/Utils.cc||323 bico
17.124

apic1# tail svc_ifc_topomgr.bin.log

Fabric topology discovery, topology and path views, state of the fabric elements.
 934||16-04-14
08:43:26.956-07:00||stimulus__||DBG4||fr=ifc_topomgr:0:0:0:0:0:0,to=ifc_to
pomgr:0:0:0:0:19:1,co=doer:19:1:0x98000000003ed349:1,si=0xb4385aa9:0x0||(e
nvelope
0x3000000f6152b: RECEIVE-SINGLE:REQUEST[stimulus/]) CONTENT
:||../common/src/framework/./core/proc/Stimulus.cc||895
954||16-04-14

09:03:24.622-07:00||ifm||DBG4||fr=ifc_policyelem:1:102:5:0,co=ifm||received
BYE from peer, msgid 0x3ba89df3d1c0, reason
0x01||../common/src/ifm/./Protocol.cc||941 bico 26.956

apic1# tail svc_ifc_vmmmgr.bin.log

Handling of endpoint attachment and moves, vCenter and vSwitch interactions.

1623||16-04-18
14:43:25.380-07:00||ifc_vmmmgr||INFO||co=doer:6:3:0x30000000004e5993:7,dn=
'"DlQgAAAcAVk13YXJlAAwAVDAxLXZDZW50ZXIADABUMDEtdkNlbnRlcgAJAGhvc3QtMTE1AAc
Adm1uaWMxAA=="',fn=[postExplicitCb]||Worker
msg for OperIssues: set - 80 , unset - 0 , Issues -
80||../svc/vmmmgr/src/gen/ifc/beh/imp/./comp/HpNicBI.cc||257
1623||16-04-18

14:43:25.380-07:00||mit_update||INFO||co=doer:6:3:0x30000000004e5993:14,dn
='"DlQgAAAcAVk13YXJlAAwAVDAxLXZDZW50ZXIADABUMDEtdkNlbnRlcgAJAGhvc3QtMTE1AA
cAdm1uaWMxAA=="'||MIT
MODIFY InstanceId:
1120:457473||../common/src/framework/./core/mo/Changer.cc||731
1624||16-04-18

14:43:25.380-07:00||replicator||INFO||co=repl:6:3:0x30000000004e5993:6||Cr
eating
replication tx 0x30000000004e5993 prev tx
0x30000000004e5992||../controller/common/src/replicator/ReplicationTx.cc||
57

Firmware / Upgrade Logs

apic1# ls /data2/log.lastupgrade
ad-healthy-scratch.log configconv.28417.log.PRESERVED configconv.28678.log.stderr svc_ifc_dbgr.bin.log.136.gz
ad-healthy-scratch.log.1.gz configconv.28417.log.stderr configconv.28680.log svc_ifc_dbgr.bin.log.137.gz
ad-healthy-scratch.log.10.gz configconv.28419.log configconv.28680.log.1 svc_ifc_dbgr.bin.log.138.gz
ad-healthy-scratch.log.13.gz configconv.28419.log.1 configconv.28680.log.PRESERVED svc_ifc_dbgr.bin.log.139.gz
ad-healthy-scratch.log.14.gz configconv.28419.log.PRESERVED configconv.28680.log.stderr svc_ifc_dbgr.bin.log.140.gz
ad-healthy-scratch.log.15.gz configconv.28419.log.stderr configconv.28683.log svc_ifc_dbgr.bin.log.141.gz
ad-healthy-scratch.log.16.gz configconv.28420.log configconv.28683.log.1 svc_ifc_dbgr.bin.log.142.gz
ad-healthy-scratch.log.17.gz configconv.28420.log.1 configconv.28683.log.PRESERVED svc_ifc_dbgr.bin.log.143.gz
ad-healthy-scratch.log.18.gz configconv.28420.log.PRESERVED configconv.28683.log.stderr svc_ifc_dbgr.bin.log.144.gz
ad-healthy-scratch.log.19 configconv.28420.log.stderr configconv.28685.log svc_ifc_dbgr.bin.log.145.gz
ad-healthy-scratch.log.PRESERVED configconv.28422.log configconv.28685.log.1 svc_ifc_dbgr.bin.log.146.gz
ad-healthy-scratch.log.stderr configconv.28422.log.1 configconv.28685.log.PRESERVED svc_ifc_dbgr.bin.log.147.gz
ad-telemeter.log configconv.28422.log.PRESERVED configconv.28685.log.stderr svc_ifc_dbgr.bin.log.148.gz
ad-telemeter.log.1 configconv.28422.log.stderr configconv.28687.log svc_ifc_dbgr.bin.log.149.gz

apic1# less /firmware/pre_upgrade_info

[main]
startVer = 1.2(1k)
endVer = 1.2(2g)
isDowngrade = False

apic1# less /firmware/insieme_installer.log

2014-10-10 23:34:35,895 INFO Attempting to unlock encrypted bootflash
2014-10-10 23:34:35,895 INFO umount /tmp/bootflash
2014-10-10 23:34:35,900 ERROR umount: /tmp/bootflash: not found
2014-10-10 23:34:35,900 INFO umount /tmp/flashenc
2014-10-10 23:34:35,906 ERROR umount: /tmp/flashenc: not found
2014-10-10 23:34:35,906 INFO mkdir -p /tmp/bootflash
2014-10-10 23:34:35,910 INFO
2014-10-10 23:34:35,910 INFO mkdir -p /tmp/flashenc
2014-10-10 23:34:35,915 INFO
2014-10-10 23:34:35,915 INFO mount -t ext4 /dev/sda1 /tmp/bootflash
2014-10-10 23:34:35,920 INFO

apic1# less /firmware/dataconv.log

Data conversion logfile
Wed Dec 24 09:20:09 UTC 2014 : ***********************************************************
Wed Dec 24 09:20:09 UTC 2014 : Started data conversion
Wed Dec 24 09:20:09 UTC 2014 : ***********************************************************
Wed Dec 24 09:20:09 UTC 2014 : Starting xinetd
Wed Dec 24 09:20:09 UTC 2014 : Removing /data/.convert
Wed Dec 24 09:20:09 UTC 2014 : Removing /data/dbserialized /data/dbdeserialized
Wed Dec 24 09:20:09 UTC 2014 : Found 0 journal files
Wed Dec 24 09:20:09 UTC 2014 : Deleting any journal files/uncommitted transactions
Wed Dec 24 09:20:09 UTC 2014 : Found 0 ser.bin files
Wed Dec 24 09:20:09 UTC 2014 : Deleting any ser.bin files
Wed Dec 24 09:20:09 UTC 2014 : Copying shmetafiles
Wed Dec 24 09:20:09 UTC 2014 : Killing nginx leftovers
Wed Dec 24 09:20:09 UTC 2014 : Changing ownership of all DB files to ifc:root
Wed Dec 24 09:20:09 UTC 2014 : Remove log settings
Wed Dec 24 09:20:09 UTC 2014 : Save up log files
Wed Dec 24 09:20:10 UTC 2014 : Getting securedb
Wed Dec 24 09:20:10 UTC 2014 : Found ENC files in /securedata/db
Wed Dec 24 09:20:11 UTC 2014 : Found ENC files in /data/db

apic1# less /firmware/dataconv_detail.log

Wed Dec 24 09:20:09 UTC 2014 : **************** Starting data conversion ******************
++ source /mgmt/env.sh
+++ export LD_LIBRARY_PATH=/mgmt/usr/lib64:/mgmt/usr/lib:/usr/lib64:/lib64
+++ LD_LIBRARY_PATH=/mgmt/usr/lib64:/mgmt/usr/lib:/usr/lib64:/lib64
+++ export PYTHONPATH=/mgmt/usr/lib64:/mgmt/usr/lib:/mgmt/opt/controller/ishell/ct64:/mgmt/opt/controller/ishell:/mgmt/opt/controller/ishell/rea
dline64:/mgmt/ishell/trinity:/mgmt/ishell:/mgmt/support/insieme
+++ PYTHONPATH=/mgmt/usr/lib64:/mgmt/usr/lib:/mgmt/opt/controller/ishell/ct64:/mgmt/opt/controller/ishell:/mgmt/opt/controller/ishell/readline6
4:/mgmt/ishell/trinity:/mgmt/ishell:/mgmt/support/insieme
+++ export MANPATH=/usr/man:/usr/share/man:/usr/local/man:/usr/local/share/man:/usr/X11R6/man:/mgmt/ishell/cli/man
+++ MANPATH=/usr/man:/usr/share/man:/usr/local/man:/usr/local/share/man:/usr/X11R6/man:/mgmt/ishell/cli/man
+++ export PAM_SERVER=/data/socket/pam
+++ PAM_SERVER=/data/socket/pam
+++ export PYEXE=/usr/bin/python2.7
+++ PYEXE=/usr/bin/python2.7
+++ export 'PASTER=/usr/bin/paster serve'
+++ PASTER='/usr/bin/paster serve'
+++ export MOROOT=
+++ MOROOT=
apic1# less /firmware/last_install_summary.1
[main]
installId = None
startEpochMs = 1412998522000
endEpochMs = 1412998837000
/firmware/last_install_summary.1 (END)