Troubleshooting Router Interfaces and Modules Issues: Document ID: 71820
Troubleshooting Router Interfaces and Modules Issues: Document ID: 71820
Issues
Document ID: 71820
Contents
Introduction
 Prerequisites
    Requirements
    Components Used
    Conventions
 Background Information
 Basic Troubleshooting Steps
    Modules Not Recognized
    Routers and Supported Modules
 Advanced Integration Modules
    AIM Slots Clock Source Issues
    AIMVPN/SSL Not Recognised
    IMA Minimum Link Not Working
    IMA Interface Flaps
 Multiflex Trunk Voice and WAN Interface Cards
    VWIC1MFTT1, VWIC2MFTT1 Hardware Installation
    VWIC21MFTT1/E1, VWIC22MFTT1/E1 Hardware Installation
    LCV and PCV Errors
 Network Modules
Prerequisites
Requirements
There are no specific requirements for this document.
Components Used
The information in this document is based on the Cisco 2600, 2800, 3600, 3700, 3800, and 7200 Series
routers.
The information in this document was created from the devices in a specific lab environment. All the devices
used in this document started with a cleared (default) configuration. If your network is live, make sure that
you understand the potential impact of any command.
Conventions
Refer to Cisco Technical Tips Conventions for more information on document conventions.
Background Information
Most of the Cisco routers are modular access devices that contain a number of slots to connect the various
network modules. These are the common issues with the router modules:
In this document, you can see basic steps to troubleshoot, identify, and isolate the module issues. You can also
see the most common issues and their solutions.
       In order to verify hardware information for the network device, use the show diag command in EXEC
        or privileged EXEC mode.
WIC Slot 1:
        Unknown WAN daughter card
        WIC module not supported/disabled in this slot
        Hardware Revision         : 3.0
        Part Number               : 73579703
        Board Revision            : A0
        Deviation Number          : 00
        Fab Version               : 02
        PCB Serial Number         : FOC07160WZ2
        RMA Test History          : 00
        RMA Number                : 0000
        RMA History               : 00
        Top Assy. Part Number     : 8000931103
        Connector Type            : 01
        Chassis MAC Address       : 0004.dd0d.798c
        MAC Address block size    : 1
        Product (FRU) Number      : PAA24T1CT3ATM=
        EEPROM format version 4
        EEPROM contents (hex):
          0x00: 04 FF 40 00 39 41 03 00 82 49 16 A5 03   42   41   30
          0x10: 80 00 00 00 00 02 02 C1 8B 46 4F 43 30   37   31   36
          0x20: 30 57 5A 32 03 00 81 00 00 00 00 04 00   C0   46   03
          0x30: 20 00 24 5F 03 05 01 C3 06 00 04 DD 0D   79   8C   43
          0x40: 00 01 FF FF FF FF FF FF FF FF FF FF FF   FF   FF   FF
          0x50: FF FF FF FF FF FF FF FF FF FF FF FF FF   FF   FF   FF
          0x60: FF FF FF FF FF FF FF FF FF FF FF FF FF   FF   FF   FF
          0x70: FF FF FF FF FF FF FF FF FF FF FF FF FF   FF   FF   FF
        WIC Slot 2:
        T1 (2 port) MultiFlex Trunk (Drop&Insert) WAN   daughter card
        Hardware revision 1.0           Board revision   B0
        Serial number     22688207      Part number      8000461402
        FRU Part Number     VWIC2MFTT1DI=
        Test history      0x0           RMA number       000000
          Connector type    PCI
          EEPROM format version 1
          EEPROM contents (hex):
            0x20: 01 24 01 00 01 5A 31 CF 50 12 06 02 00 00 00 00
            0x30: 58 00 00 00 01 01 18 00 FF FF FF FF FF FF FF FF
Slot 1:
          High Density Voice Port adapter
          Port adapter is disabled
          Port adapter insertion time unknown
          EEPROM contents at hardware discovery:
          Hardware Revision        : 1.1
          Top Assy. Part Number    : 8000356701
          Board Revision           : G0
          Deviation Number         : 031106
          Fab Version              : 02
          PCB Serial Number        : JAB0613089J
          RMA Test History         : 00
          RMA Number               : 0000
          RMA History              : 00
          Product (FRU) Number     : NMHDV=
          EEPROM format version 4
          EEPROM contents (hex):
            0x00: 04 FF 40 00 CC 41 01 01 C0 46 03   20   00   0D   EF   01
            0x10: 42 47 30 80 00 00 79 82 02 02 C1   8B   4A   41   42   30
            0x20: 36 31 33 30 38 39 4A 03 00 81 00   00   00   00   04   00
            0x30: FF FF FF FF FF FF FF FF FF FF FF   FF   FF   FF   FF   FF
            0x40: FF FF FF FF FF FF FF FF FF FF FF   FF   FF   FF   FF   FF
            0x50: FF FF FF FF FF FF FF FF FF FF FF   FF   FF   FF   FF   FF
            0x60: FF FF FF FF FF FF FF FF FF FF FF   FF   FF   FF   FF   FF
            0x70: FF FF FF FF FF FF FF FF FF FF FF   FF   FF   FF   FF   FF
Slot 2:
          4 PORT Voice PM for MARs Port adapter
          Port adapter is analyzed
          Port adapter insertion time unknown
          EEPROM contents at hardware discovery:
          Hardware revision 1.1           Board revision       B0
          Serial number     10379472      Part number          8000249102
          FRU Part Number     NM2V=
          Test history      0x0           RMA number           000000
          EEPROM format version 1
          EEPROM contents (hex):
            0x00: 01 65 01 01 00 9E 60 D0 50 09 BB 02 00       00 00 00
            0x10: 58 00 00 00 98 09 13 17 FF FF FF FF FF       FF FF FF
            0x20: FF FF FF FF FF FF FF FF FF FF FF FF FF       FF FF FF
            0x30: FF FF FF FF FF FF FF FF FF FF FF FF
          VIC Slot 0:
          FXS Voice daughter card (2 port)
          Hardware revision 1.1            Board revision      F0
          Serial number     16172601       Part number         8000249301
          FRU Part Number     VIC2FXS=
          Test history      0x0            RMA number          000000
          Connector type    Wan Module
          EEPROM format version 1
          EEPROM contents (hex):
            0x20: 01 0E 01 01 00 F6 C6 39 50 09 BD 01 00       00 00 00
            0x30: 78 0E F2 00 99 10 11 01 FF FF FF FF FF       FF FF FF
                         VIC Slot 1:
                         FXO Voice daughter card (2 port)
                         Hardware revision 1.1            Board revision           K0
                         Serial number     27584010       Part number              8000249501
                         FRU Part Number     VIC2FXO=
                         Test history      0x0            RMA number               000000
                         Connector type    Wan Module
                         EEPROM format version 1
                         EEPROM contents (hex):
                           0x20: 01 0D 01 01 01 A4 E6 0A 50 09 BF 01 00            00 00 00
                           0x30: A0 00 00 00 02 05 09 01 FF FF FF FF FF            FF FF FF
       If the module's hardware information is not shown in the show diag, reseat the hardware module. You
        can also install the module in a different slot. Some modules are not hot swappable; reload the router
        after you install the module.
       If the module's hardware information is shown in the show diag but you are not able to see the
        module interfaces in the show run or show IP interface brief output, check the hardware installation
        documentation. Some modules need router configuration after the installation.
       In the show diag output, note the error message under the WIC 0 in the Slot 0. It is displayed as
        "Unknown WAN daughter card" "WIC module not supported/disabled in this slot." This could be an
        incompatible hardware issue. Check the Routers and Supported Modules section to make sure that the
        module is supported by the router; also check the supported IOS version for that module.
Note: When you cold reboot the router, Cisco recommends that you wait at least 20 seconds before you turn
on the device.
       ATM AIM called AIMATM: The AIMATM provides asynchronous transfer mode (ATM) services
        to the WAN.
       Voice AIM called AIMVOICE30: The AIMVOICE30 provides digital signal processor (DSP)
        services, which can support up to 30 mediumcomplexity or 16 highcomplexity voice channels.
       Voice + ATM AIM called AIMATMVOICE30: The AIMATMVOICE30 combines the
        features from the AIMATM and AIMVOICE30 modules onto a single AIM module
       Voice Mail AIM called AIMCUE: The AIMCUE provides the Cisco Unity Express Voice
        Mailboxes for the users.
       VPN and SSL AIM called AIMVPN/SSL2 or AIMVPN/SSL3: The AIMVPN/SSL optimizes
        the Cisco Integrated Services Router platforms for virtual private networks in both IP Security (IPSec)
        and Secure Sockets Layer (SSL) Web and VPN deployments.
In this section, you will see some of the common issues related to the voice modules and interface cards. One
of the important steps in order to isolate the issue between the router modules and the line is to run the
Loopback test. Read the Loopback Tests for T1/56K Lines to test the circuit with the loopback test.
Note: The minimum IOS feature set required by AIMATM is IOS Plus. See the requirements and support in
the AIMATM, AIMVOICE30, and AIMATMVOICE30 modules.
Note: Compact flash can be installed only into an AIMVPN module and not into an AIMATM card.
In order to resolve this issue, configure the router to derive the clock from one source. If the service provider
furnishes the clocking, configure all the ports to source the clock from the line. In the case of ATM
pointtopoint, the service provider will not furnish the clocking. In this case, configure one end to source the
clock internally, and configure the other end to source the clock from line.
The output of the show crypto engine config command is useful to determine the cause of the problem.
This is an example of the show crypto engine config command output with the incorrect Cisco IOS
Software installed:
This is an example of a poorly seated AIM (all configuration hex values show 0):
        Configuration: 0x000000000000000000000000
        : 0x000000000000000000000000
        : 0x000000000000000000000000
        : 0x000000000000000000000000
        CryptIC Version: 000.000
        CGX Version: 000.000
        CGX Reserved: 0x0000
        PCDB info: 0x0000 0x0000 0x0000
        Serial Number: 0x0000000000
        : 0x0000000000
        DSP firmware version: 000.000
        DSP Bootstrap Version: 000.000
        DSP Bootstrap Info: 0x0000
        Compression: No
        3 DES: Yes
        Privileged Mode: 0x0000
        Maximum buffer length: 4096
        Maximum DH index: 0470
        Maximum SA index: 0940
        Maximum Flow index: 1880
        Maximum RSA key size: 0000
       Refer to the Software Support for Hardware (registered customers only) section of the Software
        Advisor (registered customers only) to ensure that the Cisco IOS Software version is compatible
        with the AIM. Also check whether a Data Encryption Standard (DES) or a Triple Data Encryption
        Standard (3DES) image is loaded. The image name will include either "56i" for DES or "k9" for
        3DES. This is an example of a DES image; c2600js56imz.1215.T9.bin is a DES image.
       Try to reseat the AIM. For complete installation instructions, refer to AIM Installation Quick Start
        Guide: Cisco 2600, 3600, and 3700 Series.
       In order to verify that the card is now correctly detected, issue the show crypto engine config
        command. The crypto engine type line will read hardware, and the configuration field will contain
        valid hexadecimal numbers. This is a sample command output for an AIM:
                Configuration: 0x000109010F00F00784000000
                : 0xA2112AB1AB68BA9C3992D377
                : 0x295801AF4A12EFD108000300
                : 0x00000000D78312B12546464B
                CryptIC Version: 001.000
                CGX Version: 001.009
                CGX Reserved: 0x000F
                PCDB info: 0x07F0 0x0084 0x0000
                Serial Number: 0x11A2B12A68AB9CBA9239
                : 0x77D35829AF01124AD1EF
                DSP firmware version: 000.008
                DSP Bootstrap Version: 000.003
                DSP Bootstrap Info: 0x0000
                Compression: No
                3 DES: Yes
                Privileged Mode: 0x0000
                Maximum buffer length: 4096
                Maximum DH index: 0470
                Maximum SA index: 0940
                Maximum Flow index: 1880
                Maximum RSA key size: 0000
Typically, you see log messages which indicate that the interface and Permanent Virtual Circuit (PVC) has
flapped, as shown.
Errors that report Slip Secs can be seen if you issue the show controllers T1 or show controllers E1
command.
        E1 0/1 is up.
          Applique type is Channelized E1  balanced
          No alarms detected.
          alarmtrigger is not set
          Version info Firmware: 20020812, FPGA: 11
          Framing is CRC4, Line Code is HDB3, Clock Source is Line.
          Data in current interval (363 seconds elapsed):
             0 Line Code Violations, 0 Path Code Violations
             85 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins
             85 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs
          Total Data (last 88 15 minute intervals):
             1 Line Code Violations, 2536 Path Code Violations,
             18319 Slip Secs, 86 Fr Loss Secs, 1 Line Err Secs, 0 Degraded Mins,
             18319 Errored Secs, 1 Bursty Err Secs, 0 Severely Err Secs, 87 Unavail Secs
In order to resolve this issue, configure a network clock source and participation for each WIC and AIM
module that is installed. Packet voice and video are sensitive to time delays. In order to prevent mismatches
and data slips, synchronize data flows to a single clock source, known as the network clock. When a network
clock is configured on a gateway, the router externally clocks one T1 or E1 port, and then passes that clock
signal across the backplane to another T1 or E1 port on another WIC or network module slot. Use of a
network clock on a gateway is configured if you name the network modules and interface cards that
participate in network clocking, and then select a controller to act as the source of timing for the network
clock.
The network clock provides timing from the source through the controller to the AIM, and then out to all the
router slots that participate. The Cisco 2600 series routers support one internal AIM slot, and the Cisco 3660
routers support two internal AIM slots. An AIM slot connects to both the main system bus and a secondary
timedivision multiplexing (TDM) bus that runs between the network module slots and, on the Cisco 2600
series, to the WAN interface card (WIC) slots.
For example, on a 2600 series switch with an AIMATM module and two VWICs used for IMA, configure
this way:
        networkclockparticipate wic 0
        networkclockparticipate wic 1
        networkclockselect 1 T1 0/0
On the 2611, 2621, and 2651, the networkclockparticipate aim 0 is not required or supported because these
routers have only one AIM slot. On the 2691, 3600, or 3700, you must also include the
networkclockparticipate aim X command for each installed AIMATM module. For more information,
refer to AIMATM, AIMVOICE30, and AIMATMVOICE30 for the Cisco 2600 Series and Cisco
3660.
It can be used as a standard WAN interface card like other Serial interface cards.
     1. Run the Loopback Test to isolate the issue between the router module and the line.
     2. Have the service provider test the line.
     3. Verify that your IOS version does not hit the current bug CSCsb00129 (registered customers only) .
Network Modules
NM1T3/E3 Installation Issues (DS3 Card)
By default, the T3 controller does not show up in the show runningconfig output. Use show version or
show diag in order to see the card, which does not show up in the show run or show interfaces output.
        Router3745#show version
        Cisco Internetwork Operating System Software
        IOS (tm) 3700 Software (C3745IK9SM), Version 12.3(12b), RELEASE SOFTWARE (fc2)
        Technical Support: http://www.cisco.com/techsupport
        Copyright (c) 19862005 by cisco Systems, Inc.
        Compiled Thu 31Mar05 18:07 by jfeldhou
        Image textbase: 0x60008AF4, database: 0x61E20000
        A summary of U.S. laws governing Cisco cryptographic products may be found at:
        http://www.cisco.com/wwl/export/crypto/tool/stqrg.html
        cisco 3745 (R7000) processor (revision 0.0) with 249856K/12288K bytes of memory.
        Processor board ID
        R7000 CPU at 350MHz, Implementation 39, Rev 3.3, 256KB L2, 2048KB L3 Cache
        Bridging software.
        X.25 software, Version 3.0.0.
        SuperLAT software (copyright 1990 by Meridian Technology Corp).
        2 FastEthernet/IEEE 802.3 interface(s)
        1 Subrate T3/E3 ports(s)
        DRAM configuration is 64 bits wide with parity disabled.
        151K bytes of nonvolatile configuration memory.
        62592K bytes of ATA System CompactFlash (Read/Write)
You need to configure the router in order to recognize the card. This configuration example shows you how to
configure the NM1T3/E3 card. Refer to the hardware installation guide Configure the Card Type and
Controller for T3 for detailed configuration information.
This occurs because the Inline Power Supply is required to provide PoE capabilities in these routers. The
external power supply option cannot be used with the Cisco 2800 or 3800 Series. The internal router power
supply can be swapped out for a new power supply with PoE capabilities if PoE is required. Examples of PoE
enabled power supplies include PWR2811ACIP=, PWR282151ACIP=, PWR3825ACIP=, and
PWR3845ACIP=. For detailed explanations and requirements, read the Cisco EtherSwitch Network
Modules.
The Hardware Installation guide shows how to replace the power supply in the Cisco 2800 and 3800 Series
Routers.
       Installing and Upgrading Internal Modules in Cisco 2800 Series Routers  Replacing the Power
        Supply
       Installing a Power Supply in a Cisco 3825 Router
       Installing a Power Supply in a Cisco 3845 Router
       T1 Troubleshooting
        Troubleshooting Serial Line Problems
        Loopback Tests for T1/56K Lines
Related Information
        Hardware Troubleshooting Index Page
        Troubleshooting Bus Error Crashes
        Troubleshooting Router Crashes
        Troubleshooting Router Hangs
        Cisco Interfaces and Modules Product Support
        Technical Support & Documentation  Cisco Systems