Alcatel-Lucent OmniSwitch 10K - Advanced Troubleshooting

OmniSwitch OS6860/OS6900/OS10K Troubleshooting Guide Part No.032996-00 Rev.A
AOS Release 7.X and 8.X January 2015
Alcatel-Lucent Page 12 of 148
2.4. Advanced Troubleshooting
Troubleshooting High CPU utilization
1. First, identify which CPU is excessively high.
CLI shell:
6860->show health
6860->show health all cpu
Maintenance shell( AOS7 & 8 are Linux based, typing su in CLI will lead to Linux BASH shell):
Top
In the maintenance shell the “top” command is used to continuously monitor the tasks consuming
CPU.
->top
Mem: 1172224K used, 849676K free, 0K shrd, 548K buff, 742380K cached
CPU: 0.0% usr 4.5% sys 0.0% nic 95.4% idle 0.0% io 0.0% irq 0.0% sirq
Load average: 0.23 0.19 0.15 2/237 3857
PID PPID USER STAT VSZ %VSZ CPU %CPU COMMAND
3857 3854 root R 2940 0.1 0 4.5 top
2090 813 root S 301m 15.2 0 0.0 /bin/ipnid
2022 813 root S 299m 15.1 1 0.0 /bin/bcmd -p
2081 813 root S 296m 14.9 1 0.0 /bin/slNi
2105 813 root S 277m 14.0 0 0.0 /bin/ipmsni
2102 813 root S 277m 14.0 0 0.0 /bin/qosnid
1749 813 root S 276m 14.0 0 0.0 /bin/qoscmmd
2069 813 root S 268m 13.6 0 0.0 /bin/stpNi
2063 813 root S 266m 13.4 0 0.0 /bin/lacpNi
2. The most common causes for high CPU utilization:
i. An abnormal process
 A process goes into an infinite loop. This is probably a software issue.
 A process is doing extensive calculations. It is possible that the network is not well
scaled.
 AOS is under a DoS attack.
ii. Abnormal traffic
 Too many messages exchanged between AOS subsystems. Examples include: extensive
logging, MAC learning, malfunctioning bus, HW interrupts.
 Too many frames or packets are trapped to CPU
Proc
Task
Proc
Task
aaaCmm
AAA
mvrpNi
MVRP NI
agCmm
Access Guardian CMM
ntpd
NetworkTime Protocol
agNi
Access Guardian NI
ofcmmd
Open Flow CMM
appMonCmm
app-mon
ofnid
Open Flow NI
appMonNiStub
app-mon
pmCmm
Port mapping CMM
bcd
Broadcom driver
pmNi
Port mapping NI
bcmd
BCM
pmmCmm
Port Mirroring & Monitoring
CMM
bfd
BFD CMM
pmmnid
Port Mirroring & Monitoring Ni