Site

TIMIŞOARA
Originator
TC Troubleshooting Guide Release B11
TC Team

Domain : ALCATEL-LUCENT 9153/BSS

Division :
Rubric : TC
Type : Troubleshooting Guide
Distribution Codes Internal: External :
All rights reserved. Passing on and copying of this
document, use and communication of its contents

Category: TROUBLE SHOOTING GUIDE

PREDISTRIBUTION:

TC B11 DPM : A. Negrescu

TC : Test Team, L3Support

VAL : Loic Raffray, Mihai Brateanu

ABSTRACT
This document provides the operational workarounds, which can be applied by the Alcatel-
Lucent teams on TC B11 from the first version.

Approvals
Name A.Negrescu
App. DPM/TC/B11

ED01 RELEASE01 TC Troubleshooting Guide B11 Release

Site 3BK 29873 MAAA PWZZA 1/27
 REVIEW
Ed. 01 Proposal 01 16/10/2008 TD/MRC/TC/2008.543

HISTORY
Ed. 01 Proposal 01 16/10/2008 First proposal
Ed. 01 Proposal 02 23/11/2008 Second proposal
Update: 3.4;3.5 paragraph
New paragraph added: 2.1
Ed. 01 Proposal 03 23/11/2009 Third proposal
New paragraph added: 1.1; 1.2; 1.3;
3.3; 3.4; 3.5; 4.2; 4.3; 4.4; 4.6; 5.2;
5.4; 5.5; 5.6; 5.7
All rights reserved. Passing on and copying of this
document, use and communication of its contents

ED01 RELEASE01 TC Troubleshooting Guide B11 Release

Site 3BK 29873 MAAA PWZZA 2/27
 TABLE OF CONTENT

SCOPE ........................................................................................................................................................4
1 BASIC PROCEDURES .......................................................................................................................5
1.1 Not possible to connect with terminal using MMI to an XB Board.............................................5
1.2 [TCIL]After MT120 NB software activation, return to factory software and flash corruption .....6
2 OMC/TC COMMUNICATION ..............................................................................................................7
2.1 What to do in case of message “SNMP error during TC declaration –couldn’t obtain all
needed information from TC board” ............................................................................................................7
3 TCIF Administration.............................................................................................................................8
3.1 Can not read IPTC2 registers because of "SIGALRM received during IOCTL segmentation
fault"……… ..................................................................................................................................................8
3.2 The IP board crashed after a call. .............................................................................................9
All rights reserved. Passing on and copying of this
document, use and communication of its contents

3.3 BTP - New ports to simplify firewall configuration. ....................................................................9

3.4 Port 5555 used for collecting traces on TCIFI boards. ............................................................10
3.5 The TC G2.5 IP is not working with one TCIF board...............................................................13
3.6 [B11-TLO-FR] IP mode : after performing the 3 flashes and reboot the TCIF, the board
cannot restart normally due to "Loading STM1" frozen on Serial link. ......................................................14
3.7 Upgrade the IP daughter board ...............................................................................................15
3.8 How to check which board is giving the clock to TCIFI ...........................................................18
4 MT120 Administration .......................................................................................................................19
4.1 [Val-Vz]- TC-AUDIT-NEEDED sent in loop after switchover TCIF from slot#1 to slot#0. .......19
4.2 [HSI] MT120 Legacy boards connected to the standby TCIF, i.e. boards missing from TC-
NEM's "Rack Overview" window.
After an unsuccessful TCIF take over , all legacy MTs from the rack
are connected to stand-by TCIF
[TCIL to HSI] after switching all boards in HSI, some boards are on
the wrong TCIF. .........................................................................................................................................20
4.3 [B11-TLO-FR] TDM mode: During MT120 software download from AZ03C to AZ03D the TC
configuration is lost. ...................................................................................................................................20
4.4 TC is reporting incorrect values for the capability of MT120 in case of Legacy (0) and WB(2)
boards…….................................................................................................................................................20
4.5 How to collect traces on MT boards. .......................................................................................21
4.6 [HSI][xB] When performing MT120 SWR trough TCIF it is possible that a MT120 xB board
becomes misaligned (red) in TC-NEM's "Rack Overwiev" with the "Reason: File List Report Failure"....22
4.7 [HSI]Sporadically MT120 board continuously connect/disconnect to TCIF ............................23
4.8 How to activate the factory software on legacy board.............................................................23
5 TC-NEM Administration.....................................................................................................................24
5.1 TC NEM installed on a Windows Vista PC rejects the Local setting although a local
connection is set between TC NEM and TCIF ..........................................................................................24
5.2 Sometimes one or more processes from TC-NEM remain active in Windows and need to be
close manually from Task Manager, in order to be able to restart TC-NEM again. ..................................24
5.3 The TCIF window is not opened when connect the TC NEM to TCIF ....................................25
5.4 Changing the Management Mode for a MT120 Legacy or MT120 xB board from HSI to TCIL
or TCIL to HSI generates a timeout...........................................................................................................25
5.5 Sometimes, in TC-NEM it is not possible to see any alarm about MT boards after a request26
5.6 How to save the TC-NEM log ..................................................................................................26
5.7 Sometimes when performing MT120 replacement with the Wizard provided by TC-NEM it is
possible that the application remains hanging. .........................................................................................27

ED01 RELEASE01 TC Troubleshooting Guide B11 Release

Site 3BK 29873 MAAA PWZZA 3/27
 RELATED DOCUMENTS:
• OMC3 B11 Troubleshooting Guide 3BK 29007 LAAA PWZZA
• MXBSC TROUBLESHOOTING GUIDE B10 3BK112061093PWZZA
All rights reserved. Passing on and copying of this
document, use and communication of its contents

SCOPE

The aim of this document is to provide hints and workarounds to operator using the B11 TC.
This document is restricted for ALCATEL-LUCENT intern usage, notably for ALCATEL-
LUCENT personnel providing on site support at customer premises.
The chapters BASIC PROCEDURES and the TC ADMNISTRATION are the stable basic first
aids, the other chapters are supposed to be updated each time new problems will occur.

ED01 RELEASE01 TC Troubleshooting Guide B11 Release

Site 3BK 29873 MAAA PWZZA 4/27
 1 BASIC PROCEDURES

1.1 Not possible to connect with terminal using MMI to an XB Board

Reference FR
None

Problem description
The connection to the xB board is established from TC-NEM using MMI. A cross cable is used to
connect the PC on which the TC-NEM is installed, to the xB board on the Ethernet port. For some
boards the connection is possible but for the some others not
The problem is generated by the facts that the XB boards have the IP addresses in different networks
All rights reserved. Passing on and copying of this
document, use and communication of its contents

Corrective action

On the debug trace of the board tape “Enter” and then “ifconfig”. The answer of the board will be like
the following one :
“lo0: flags=48049<UP,LOOPBACK,RUNNING,MULTICAST,INET_UP> mtu 1536
inet 127.0.0.1 netmask 0xff000000
qefcc0: flags=78043<UP,BROADCAST,RUNNING,MULTICAST,NOTRAILERS,INET_UP> mtu 1500
inet 10.0.0.201 netmask 0xffffff00 broadcast 10.0.0.255
ether 00:c0:be:94:00:52
value = 0 = 0x0”

where in red and bold is the Ip address of the board

If the inet is 192.168.255.249, then check if the network card of the PC is configured with the address
192.168.255.250 and the mask 255.255.255.0.
In the "Settings" window from TC-NEM, in the section for MMI enter the following IP: 192.168.255.249
and the port 2005. In this case the board has the IP: 192.168.255.249

If the inet is 10.0.0.201 then configuration for the PC running TC-NEM: 10.0.0.202, case in which the
board has the IP: 10.0.0.201. The mask remains the same.

And so on if the inet is different, the Ip of the computer and the IP of the board in the TC-NEM window,
should be set up accordingly

Problem solved
Not Applicable

ED01 RELEASE01 TC Troubleshooting Guide B11 Release

Site 3BK 29873 MAAA PWZZA 5/27
 1.2 [TCIL]After MT120 NB software activation, return to factory
software and flash corruption

Reference FR
3BKA37FBR260852

Problem description
Sometimes the NPSA gets corrupted and the board restarts in loop.

Corrective action
I.) Boot Selector Upgrade (the newest is MT120xB_BootSel_0203030)
All rights reserved. Passing on and copying of this
document, use and communication of its contents

Prerequisites:

- Cable with "Flash unprotect"

- Serial RS232 cable
- DTerm

Steps to be performed in order to update the Boot Selector version:

1. Connect the RS232 cable to the "Flash unprotect cable" and plug it in the debug connector of the
MT120 xB board.
[ PC --> Serial RS232 cable ---> "Flash unprotect cable" ---> MT120 xB board ]
The "Flash unprotect cable" has the lever set so, that the access to the Flash of the MT120 xB board
is unprotected.

2. Open a DTerm session. Press "CTRL"+x

3. When you see a countdown (Boot will continue in 2, 1, ...) on the debug trace press "Enter". You will
enter the command line interface, which allows you to type commands.

4. Write:

dlf "BOOT"

Press "Enter".
When you see "Please send file...", press on the Red arrow in DTerm and select the desired ".fh2"
file. You will see the file being uploaded to the board.

5. Check that the transfer has been successfully performed, when you see on the screen:

value = 0 = 0x0

6. Press "CTRL"+x .The board will reboot with the Boot Selector updated.

Perform steps from 2 to 6 for all MT120 xB boards you want to update.

II.) Download the last MT120xB Origin Software Package

The following command is entered on DTerm command line:

ED01 RELEASE01 TC Troubleshooting Guide B11 Release

Site 3BK 29873 MAAA PWZZA 6/27
 In case of serial connection for file transfer:
- Connect DTerm to the MT120 xB board over serial connection;
- Press power-on reset (button);
- Stop the MT120xB boot sequence (during the countdown);
- Enter the following command at the MT120 prompt: dlp “ORIGIN”;
- Un-protect FLASH;
- In Dterm, select the origin package to be downloaded (e.g. MTXBAX36.ECR) and wait until
the download is finished;
- Protect FLASH;
- Enter the following command at the MT120 prompt: Package_Activate 0

Problem solved
Not Applicable
All rights reserved. Passing on and copying of this
document, use and communication of its contents

2 OMC/TC COMMUNICATION

2.1 What to do in case of message “SNMP error during TC declaration –couldn’t

obtain all needed information from TC board”

Reference FR
None

Problem description
The communication between OMC-R and TC is broken. This problem could appear if the networks
where the OMC and TC are not communicating, if some preliminary configurations were not done on TC
or if an OMC problem.

Corrective action
From the OMC terminal ping the active TCIF.
- If no answer check if the networks are compatible
Using a putty connect to active TCIF:
- Check if the heartbeat and oaed process are running
Using the TC-NEM connected to the distant IP of the TC check:
- The BSC declared on OMC as associated to TC, exists in TCIF database (Rack config window)
- The hostid of the OMC-R is declared on TCIF and it is associated to the right BSC (BSC Host Ids
window)
From the OMC-R terminal as axadmin user check if on TC side there are default passwords:
axadmin@host$cd /usr/sfw/bin
axadmin@host$./ snmpget -v 3 -u omc3 -l Authpriv -x DES -X privomc3 -a MD5 -A authomc3
<TC_floating_ip_address> 1.3.6.1.4.1.637.70.6.1.3.1.5.0

Normally the TC should answer with rack number. If the TC answers then is an OMC-R problem. Else
check the TC configuration.

To reset the passwords on the TCIFI boards, so that the communication with OMC-R can be established
the following workaround has to be performed:

TCIF in slot 0: assure that the process are stopped (heartbeat, oaed, saed, hsid)

ED01 RELEASE01 TC Troubleshooting Guide B11 Release

Site 3BK 29873 MAAA PWZZA 7/27
 /optX/heartbeat-2.1.2/init.d/heartbeat stop
where “X” is 0 or 1 depending where all processes are started from.
This command stops the heartbeat process. Check that also oaed, saed, hsid are stopped.

ps –ef | grep d.c

TCIF in slot 1 : assure that the process are stopped (heartbeat, oaed, saed, hsid)

/optX/heartbeat-2.1.2/init.d/heartbeat stop
where “X” is 0 or 1 depending where all processes are started from.
This command stops the heartbeat process. Check that also oaed, saed, hsid are stopped.

ps –ef | grep d.c

Apply the next commands to both TCIFIs one by one:

All rights reserved. Passing on and copying of this
document, use and communication of its contents

cd /mnt/flash
ls –a
rm –Rf .net-snmp
reboot

After this has been done on both boards, declare the TC in OMC-R.

Problem solved
Not Applicable

3 TCIF Administration

3.1 Can not read IPTC2 registers because of "SIGALRM received during IOCTL
segmentation fault".

Reference FR
3BKA37FBR252568

Problem description
The error "SIGALRM received during IOCTL segmentation fault" is reported when FIBd table is reading
more then 3 times.

Corrective action
Reboot TCIF boards.

Problem solved
Not applicable

ED01 RELEASE01 TC Troubleshooting Guide B11 Release

Site 3BK 29873 MAAA PWZZA 8/27
 3.2 The IP board crashed after a call.

Reference FR
3BKA37FBR238106

Problem description
The read of internal tables on the JGIPTC board displays out of range value
The IPFWD driver freezes sometimes when TCIF software tries to access it (read/write).

Corrective action
Workaround: reboot TCIF.

Problem solved
Not applicable
All rights reserved. Passing on and copying of this
document, use and communication of its contents

3.3 BTP - New ports to simplify firewall configuration.

Reference CR
3BKA20CBR261193/3BKA20CBR261194

Problem description
In the IP transport to simplify the configuration of firewalls.

Corrective action
Port range changed as specified in the CR Document.

M2UA_SCTP_PORT_TC (STCP port used for M2UA protocol- signalisation to/from A interface),
TC side

minim : 52592
maxim : 52615
step size = 1

IPTCH_UDP_PORT_TC_NONMUX (UDP ports used TC-TC (eTRAUP) flow in UL)

minim : 52480
maxim : 52526
step size = 1

IPTCH_UDP_PORT_TC_MUX (UDP ports of IPTCH link used on TC side (for MUXTRAUP and
TRAUP protocols))

minim : 54528
maxim : 54575

ED01 RELEASE01 TC Troubleshooting Guide B11 Release

Site 3BK 29873 MAAA PWZZA 9/27
 Tcsl-Port-BSC/TCside (TCP Port of TCSL link)

minim : 52528
maxim: 52577

Careful! In TC_NEM/ Rack Config tab when the BSCs are declared at tcsl_Port _TC_Side is
safely to use the ports from 4 to 4 step. (eg : for BSC1 : TC_Port_Side 52528, BSC2 :
TC_Port_Side 52532, …..).
Those ports should be the same as in BSC build.

Problem solved
Not applicable
All rights reserved. Passing on and copying of this
document, use and communication of its contents

3.4 Port 5555 used for collecting traces on TCIFI boards.

Reference FR
3BKA20FBR270673

Problem description
Is needed to have server port 5555 and client port 1025..65535 unlocked on client live network in order
to take traces from TCIF.

Corrective action
Check that the ping between the Windows PC (on which the traces are saved) and TCIF it’s working.

How to collect TCIFI traces:

TCIF boards are installed in one TC G2.5 rack: one is active and the other one is standby.
The traces are saved on the Windows XP PC.
The traces on TCIF board are possible to be taken, using the current method, beginning with the
TCIFAZ01Q software version.
The logging_client_AZab software package is available on site (ab is the version).

First steps

1. Unpack the logging_client_ab.zip file on your computer (e.g. in C:/)

2. A new directory (e.g. inside C:/) will be created: logging_client_ab.
3. From this new directory, the file “run_logging_client.bat” will be used to take the traces.

Connection to the TCIF

The local serial connection from the PC to the TCIF board (RS connector) can be used.
Instead of using the local serial connection, it is possible also to perform the following commands
(chapters 3, 4 and 5) on a remote SSH (PUTTY) connection on each TCIF board.
Starting with the version Logging_client_AZ02F, it is recommended to use a remote PUTTY connection
on each TCIF board. For remote SSH connection on PUTTY, use HDEV user and cit_ln password.
For maintenance operations, change to root user, using the su command (the password for root is tcif).

ED01 RELEASE01 TC Troubleshooting Guide B11 Release

Site 3BK 29873 MAAA PWZZA 10/27
 Set the configuration file

All actions are performed using root user.

On each TCIF board is present the file “logging.conf” in the directory /opt/tcifR-1.1/share/tcif.
Initially this file should look similar to the example below:

# LOGGING PARAMETERS
####################

# oaed section
OAED_LOGGING_ENABLED=0
OAED_LOG_SERVER_HOST=192.168.89.17
OAED_LOG_SERVER_PORT=9876
OAED_ALLOW_TRACE_LEVEL=0
All rights reserved. Passing on and copying of this
document, use and communication of its contents

# hsid section
HSID_LOGGING_ENABLED=0
HSID_LOG_SERVER_HOST=192.168.89.17
HSID_LOG_SERVER_PORT=9876
HSID_ALLOW_TRACE_LEVEL=0

# saed section
SAED_LOGGING_ENABLED=0
SAED_LOG_SERVER_HOST=192.168.89.17
SAED_LOG_SERVER_PORT=9876
SAED_ALLOW_TRACE_LEVEL=0

# qmuxd section
QMUXD_LOGGING_ENABLED=0
QMUXD_LOG_SERVER_HOST=192.168.140.17
QMUXD_LOG_SERVER_PORT=9876
QMUXD_ALLOW_TRACE_LEVEL=0

Inside this file, set the value 1, instead of the value 0, for each daemon on which will be activated the
LOG file.
cd /opt/tcifR-1.1/share/tcif
vi logging.conf
daemon_LOGGING_ENABLED=1

The daemon can be oaed, hsid , saed or qmuxd. The most important trace is the oaed trace, that means
in the logging.conf file will be set:
OAED_LOGGING_ENABLED=1

In this case a low level of trace will be available, level which should be used when a full TC G2.5 rack
with 48 MT120 boards is used.

daemon_ALLOW_TRACE_LEVEL =1
In this case the high level of traces will be available.

A non valid combination is:

daemon_LOGGING_ENABLED=0
daemon_ALLOW_TRACE_LEVEL=1

ED01 RELEASE01 TC Troubleshooting Guide B11 Release

Site 3BK 29873 MAAA PWZZA 11/27
 Start the logd

All actions are performed using root user.

The logd process must be started. The procedure, to start it manually, is described in 4.1. But if the TCIF
restarts because of any reason, the logd process stops on that TCIF. It must be started again manually.
In order to avoid to start manually the logd process each time a TCIF reboots, it is necessary to perform
the actions described in 4.1 and 4.2.

Manual start logd

On each TCIF board is present the file logd process in the directory /opt/tcifR-1.1/init.d
This process must be started.

cd /opt/tcifR-1.1/init.d
./logd start
All rights reserved. Passing on and copying of this
document, use and communication of its contents

Check if the logd is running:

cd /opt/tcifR-1.1/init.d
./logd status

Automatic restart logd

In order to avoid to start manually the logd process, in case the TCIF board restarts, it is necessary to
change the file 99heartbeat.

cd /opt/tcif-scripts-1.1/etc/rc.d
vi 99heartbeat
Uncomment the line:
#su root -c "/opt${instance}/tcif?-1.1/init.d/logd start"

Save the modification.

Make all these steps on both TCIF boards.

Restart the daemon

All actions are performed using root user.

The daemon, for which is defined the trace to be taken, must be restarted.
In order to do this, it is necessary to stop/start the heartbeat process.

Important: Go further only after the heartbeat is stopped on both TCIF boards. Between the
stopping of the heartbeat on one TCIF board and the stopping of the heartbeat on the other TCIF,
there must be no more than 1 minute.

Stop the process heartbeat on both TCIF boards:

/opt/heartbeat-2.1.2/init.d/heartbeat stop

When the heartbeat process is stopped, there will be no other process running (hsid, saed, oaed,
mmirelay, qmux_forwarder, in.tftpd are automatically stopped). Check this, on each TCIF board, with:

ps –eaf | grep tcif

Start the process heartbeat on both TCIF boards:

ED01 RELEASE01 TC Troubleshooting Guide B11 Release

Site 3BK 29873 MAAA PWZZA 12/27
 /opt/heartbeat-2.1.2/init.d/heartbeat start

All the other processes will be started automatically, when the heartbeat process is started.

Check if all the processes are started again, in 3 minutes.

ps –eaf | grep tcif

Start the traces on both TCIF boards

Check that the ping between the Windows PC (on which the traces are saved) and TCIF its working.

Start the command prompt from Windows: Start/run/cmd, go to the directory where the logging client
was extracted (e.g. C:/logging_client_AX42A) and execute the command:
All rights reserved. Passing on and copying of this
document, use and communication of its contents

run_logging_client.bat IP_ETH1_TCIF1 IP_ETH1_TCIF2

where: IP_ETH1_TCIF1 and IP_ETH1_TCIF2 are he IP addresses of the two TCIF boards.

The traces will be saved in the default path .\Traces\(Board0 | Board1) – normally this two
directories exist after unpacking the client (e.g. C:/logging_client_02R/Traces/Board0). In order to save
the traces to a different location, the run_logging_client.bat file must be modified and the directory
Traces plus the two sub-directories must be created manually.

Final checks

In the traces directory, in each sub-directory (e.g. C:/logging_client_AZ02R/Traces/Board0 or

C:/logging_client_AZ02R/Traces/Board1), there will be the trace files for each daemon which was set.

Stop the traces

To stop the traces it does not suffice to stop the logd process. The modified values in the
logging.conf file must be reset to 0. After this has been done, the heartbeat process has to be stopped
and restarted.

Note
It is recommended to take only the oaed traces, because of TCIF and network overload.

Problem solved
Not applicable

3.5 The TC G2.5 IP is not working with one TCIF board.

Reference FR
3BKA20FBR251609

Problem description
TC G2.5 IP (Package 2) is not working with only one TCIF board or it is very slow.
Not possible to declare BSC at TC NEM IP connection.

ED01 RELEASE01 TC Troubleshooting Guide B11 Release

Site 3BK 29873 MAAA PWZZA 13/27
 Corrective action
Workaround: reboot TCIF.

Problem solved
Not applicable

3.6 [B11-TLO-FR] IP mode : after performing the 3 flashes and reboot the TCIF,
the board cannot restart normally due to "Loading STM1" frozen on Serial
link.

Reference FR
3BKA37FBR249426

Problem description
All rights reserved. Passing on and copying of this
document, use and communication of its contents

The problem is coming from an old version of the EPLD JGTCBM "e" instead of "f". New ICS03 TCIFI is
requested for the TCIFI HW SUB DR4.

Corrective action
An upgrade procedure for STM-1 board (TestOBC binary file, JGTCM):
JGTCBM EPLD load

It is recommended to use a remote PUTTY connection on each TCIFI board. For remote SSH
connection on PUTTY, use HDEV user and cit_ln password.
For maintenance operations, change to root user, using the su command (the password for root is tcif).

Take care, the files for the STM-1 board are already on the TCIF.
The STM-1 board needs to be on the TCIF board.

Check the version of the STM-1 board:

/opt/tcifR-1.1/bin/TestObcDriver.test Test_OBC_Drv test_ReadStm1Version

If the result is different than “f”, the STM-1 board needs to be upgraded.

Stop the process on the TCIFI boards:

/opt/heartbeat-2.1.2/init.d/heartbeat stop
/opt/tcifR-1.1/init.d /logd stop

Check that all processes are stopped (hsid, oaed, saed, tclogger):

ps –eaf | grep hsid

ps –eaf | grep oaed
ps –eaf | grep saed
ps –eaf | grep tclogger

To upgrade the JGTCBM EPLD, the following command should be entered:

- insmod /lib/modules/2.6.10_mvlcge401-8560ads/jts10_jtag_drv.ko
- cat /proc/devices|grep jtag

The command will give the “major” for the driver jts10_jtag_drv <MAJOR> jtag_drv

ED01 RELEASE01 TC Troubleshooting Guide B11 Release

Site 3BK 29873 MAAA PWZZA 14/27
 - mknod /dev/jtag_drv c <MAJOR> 0

To test the communication with the JGTCBM:

- cd /fpga
- jts10_jtag_loader Firecron_infra_lp1.bvf

JTAG JGTCBM management: Mar 18 2008-13:36:26 release 01.01

Loading file: Firecron_infra_lp1.bvf……………..

Report:
 Total Byte Number : 452
 Number of Test: 232
 Number of Error: 0
 Number of Success: 232
Load OK!!!
All rights reserved. Passing on and copying of this
document, use and communication of its contents

If the result of the Firecron_infra_lp1.bvf is Load Failed!!!, you should not try to load EPLD.

To load the JGTCBM:

- jts10_jtag_loader tcbm_v1t15.bvf

JTAG JGTCBM management : Mar 18 2008-13:36:26 release 01.01

Loading file: tcbm_v1t15.bvf………
Report:
 Total Byte Number: 747897
 Number of Test: 272008
 Number of Error:0
 Number of Success:272008
Load OK!!!!

It takes some 2 minutes to load the EPLD.

Reboot the TCIFI boards and after the TCIF is up and running, check the version of the STM-1 board:

cd /opt/tcifR-1.1/bin
TestObcDriver.test Test_OBC_Drv test_ReadStm1Version

If the result is “f”, the STM-1 was upgraded successfully.

Problem solved
Not applicable

3.7 Upgrade the IP daughter board

Reference FR
None

Problem description
The problem is coming from the soft version of the IPTC.

ED01 RELEASE01 TC Troubleshooting Guide B11 Release

Site 3BK 29873 MAAA PWZZA 15/27
 Corrective action
An upgrade procedure for IP board (sfl_altera, infra_jaxip jgiptc- binary files, JGIPTC_Load):

JGIPTC FPGA load

It is recommended to use a remote PUTTY connection on each TCIFI board. For remote SSH
connection on PUTTY, use HDEV user and cit_ln password.
For maintenance operations, change to root user, using the su command (the password for root is tcif).

Before starting the upgraded the board check the version of the IPTC:

/opt0/tcifR-1.1/bin/ipfwutil Test_IPFW_Drv test_readCustomRegister

All rights reserved. Passing on and copying of this
document, use and communication of its contents

"Enter Register Address(0xhhhhhhhh): 22:51:01.243554:813861664:TRACE: {

Input_Reactor_Device_Wrapper_Base::svc
22:51:01.243914:813861664:DEBUG:Owner was set"
introduce the value:

0x20000000

If the result is :
„Register at address 0x20000000 has value: hexa: 0x3000005 (decimal: 50331653)“

is not neccessary to upgrade the board.

If the value indicates a version older than 0x3000005, the upgrade is necessary.

If the processes are up and running:

stop the processes on both TCIFI boards:

/opt/heartbeat-2.1.2/init.d/heartbeat stop
/opt/tcifR-1.1/init.d /logd stop

Check that the oaed, saed, hsid, tclogger, are stoped.

ps –eaf

The following commands will be executed on the TCIFI:

# insmod /lib/modules/2.6.10_mvlcge401-8560ads/jts10_jtag_drv.ko

#cat /proc/devices | grep jtag

The command will give the major for the driver jts10_jtag_drv
<MAJOR> jtag_drv

#mknod /dev/jtag_drv c <MAJOR> 0

Copy 3 files .bvf (infra_jaxip.bvf, sfl_altera.bvf, jgiptc.bvf) in the /mnt/nfs directory.

ED01 RELEASE01 TC Troubleshooting Guide B11 Release

Site 3BK 29873 MAAA PWZZA 16/27
 Please be careful that all three files *.bvf to have 755 rights and user and group to be root.

#chmod 755 *.bvf

#chown root: root *.bvf

If cannot be change the rights and the owner of the three files (*.bvf) from /mnt/nfs path than copy them
in /tmp.

To test the communication with the JGIPTC:

# cd /tmp
#jts10_jtag_loader infra_jaxip.bvf

JTAG JGTCBM management : Mar 18 2008-13:36:26 release 01.01

All rights reserved. Passing on and copying of this
document, use and communication of its contents

Loading file : infra_jaxip.bvf.....

Report:
-> Total Byte Number: 1951
-> Number of Test: 518
-> Number of Error: 0
-> Number of Success: 518
Load OK !!!!

If the result of the file infra_jaxip.bvf is Load Failed !!!!, you should not try to load the FPGA.

If the result of the file infra_jaxip.bvf is Load OK :

load the JGIPTC :

# jts10_jtag_loader sfl_altera.bvf

JTAG JGTCBM management : Mar 18 2008-13:36:26 release 01.01

Loading file : sfl_altera.bvf.....

Report:
-> Total Byte Number: 6425747
-> Number of Test: 3212847
-> Number of Error: 0
-> Number of Success: 3212847
Load OK !!!!

N.B.: It takes 5 minute to load this file.

-#jts10_jtag_loader jgiptc.bvf

JTAG JGTCBM management : Mar 18 2008-13:36:26 release 01.01

Loading file : jgiptc.bvf.....

Report:
-> Total Byte Number: 41194355
-> Number of Test: 20448929
-> Number of Error: 0

ED01 RELEASE01 TC Troubleshooting Guide B11 Release

Site 3BK 29873 MAAA PWZZA 17/27
 -> Number of Success: 20448929
Load OK !!!!

N.B.: It takes 1h to load the FPGA.

On the both TCIFI boards the all processes are still stopped.
Only on the Master TCIFI board start the heartbeat process:

/opt/heartbeat-2.1.2/init.d/heartbeat start

From TC-NEM (trough IP connection – introduce the IP of the Master board) perform “lock” on Stand-by
board.
The blue led is lighting on the Stand-by board. In TC-NEM the Stand-by board became red.
Then perform “unlock” on Stand-by board.
All rights reserved. Passing on and copying of this
document, use and communication of its contents

Also for the other TCIFI board, from TC-NEM perform “lock” on Master board.
The blue led is lighting on the newly Stand-by board (a takeover occurred also).
In TC-NEM the Stand-by board became red.
Then perform “unlock” on Stand-by board.

Please check if the all processes (oaed, saed, hsid) are up and running on both TCIFI from putty
connection:

ps –eaf

Check that the upgrade was made by giving the command on both TCIFI boards:
/opt/tcifR-1.1/bin/ipfwutil Test_IPFW_Drv test_readCustomRegister

"Enter Register Address(0xhhhhhhhh): 22:51:01.243554:813861664:TRACE: {

Input_Reactor_Device_Wrapper_Base::svc
22:51:01.243914:813861664:DEBUG:Owner was set"
introduce the value „0x20000000“

If the version is the upgraded one, then the upgrade was done successfully.

Problem solved
Not applicable

3.8 How to check which board is giving the clock to TCIFI

Reference FR
None

Problem description
To verify which board is clock master.

ED01 RELEASE01 TC Troubleshooting Guide B11 Release

Site 3BK 29873 MAAA PWZZA 18/27
 Corrective action
In order to check that the TCIF has taken the synchronization from the targeted board is necessary to
give the commands:

#cd /opt/tcifR-1.1/bin
#./hsiutil Test_HSI_Drv test_GetClockRef 2>/dev/null
In the output of the execution you will have “Clock reference id xx” , where xx is the MT120 board from
which the TCIF has take the clock. The numbering is done from left to right and from bottom to top
beginning from 0.

Example:

If the output is:

“Running 1 testClock reference is 10” the MT120 board has HSI number 10 (TEI74)
All rights reserved. Passing on and copying of this
document, use and communication of its contents

Problem solved
Not applicable

4 MT120 Administration

4.1 [Val-Vz]- TC-AUDIT-NEEDED sent in loop after switchover TCIF from slot#1 to
slot#0.

Reference FR
3BKA37FBR254772

Problem description
After new switchover from slot#1 to slot#0, TCSL and TC-OM was established even if TC-AUDIT
scenario was successfully, and TC-OM established, TC continue to sent TC-AUDIT-NEEDED.

Corrective action
Plug out/in the board

Problem solved
Not applicable

ED01 RELEASE01 TC Troubleshooting Guide B11 Release

Site 3BK 29873 MAAA PWZZA 19/27
 4.2 [HSI] MT120 Legacy boards connected to the standby TCIF, i.e. boards
missing from TC-NEM's "Rack Overview" window.
After an unsuccessful
TCIF take over , all legacy MTs from the rack are connected to stand-by TCIF

[TCIL to HSI] after switching all boards in HSI, some boards are on the
wrong TCIF.

Reference FR
3BKA37FBR259302, 3BKA37FBR258710, 3BKA20FBR255919

Problem description
TCIF 1 (Slot 0) is up, but most of the MT120 Legacy boards from the rack are connected to TCIF 2 (Slot
1), after TCIF takeover, or after changing the software management mode from TCIL to HSI, with the
result that MT120 boards are missing from TC-NEM’s “Rack Overview” when connected trough IP.
All rights reserved. Passing on and copying of this
document, use and communication of its contents

Corrective action
Connect locally on the missing boards from TC-NEM's "Rack Overview" and perform a restart, or,
perform a reset of the active TCIF.

Problem solved
Not applicable

4.3 [B11-TLO-FR] TDM mode: During MT120 software download from AZ03C to
AZ03D the TC configuration is lost.

Reference FR
3BKA20FBR262363

Problem description
Part of the persistent FLASH memory is overwritten, leading either to loose of the configuration or fall-
back to origin software. No traffic possible.

Corrective action
The problem was corrected for the software with a dimension of 476 blocks.

Problem solved
Not applicable

4.4 TC is reporting incorrect values for the capability of MT120 in case of Legacy
(0) and WB(2) boards.

Reference FR
3BKA37FBR273364

ED01 RELEASE01 TC Troubleshooting Guide B11 Release

Site 3BK 29873 MAAA PWZZA 20/27
 Problem description
The TC is sending 0 (Legacy MT120), 1 (NB) or 2(for WB) which is wrong.
In B11 - The MT120 capabilities reported should be:
- 00000001(1 in dec) - for Legacy (TFO_AMR_NB)
- 00000001(1 in dec) - for xB_NB (TFO_AMR_NB)
- 00000011(3 in dec) - for xB_WB (TFO_AMR_NB and TFO_AMR_WB)

Corrective action
The correction was made accordantly with:
- 00000001(1 in dec) - for Legacy (TFO_AMR_NB)
- 00000001(1 in dec) - for xB_NB (TFO_AMR_NB)
- 00000011(3 in dec) - for xB_WB (TFO_AMR_NB and TFO_AMR_WB)

Problem solved
All rights reserved. Passing on and copying of this
document, use and communication of its contents

Not applicable

4.5 How to collect traces on MT boards.

Reference FR
None

Problem description
There are two possibilities to take MT debug traces, using Tera Term or Dterm applications.
In order to take the debug traces a "TC NEM" cable is needed but not plugged in MMI position but in
debug for both Legacy and XB board.

Corrective action
How to take debug traces using Tera Term aplication:

- The cable connected between MMI of the board and the COMx of the computer
- Open the Tera Term and select the serial connection and the COMx
- From terminal menu select "Setup" and the "Serial port"
- Select the settings :
- Port: COMx
- Data : 8 bit
- Parity : none
- Stop: 1 bit
- Flow control : none
- Transmit delay: 0
If the board is a Legacy one selected Baudrate is 38400
If the board is a XB one , the the Baud rate should be 115200
To save the debug traces select on the Tera term window the “File ” and “Log ” options and give a
name and a place were the traces to be stored

ED01 RELEASE01 TC Troubleshooting Guide B11 Release

Site 3BK 29873 MAAA PWZZA 21/27
 How to take debug traces using Dterm aplication:

- The cable connected between MMI of the board and the COMx of the computer
- Open the Dterm window
- From the terminal menu select "Settings" , then "Itf. Communication Settings"
- Select the following settings:

- Port: COMx
- Data bits : 8 bit
- Parity : none
- Stop bits: 1 bit
- Protocol: none

If the board is a Legacy one selected Baudrate is 38400

If the board is a XB one , the the Baudrate should be 115200
All rights reserved. Passing on and copying of this
document, use and communication of its contents

After the settings done the push the RS232 button from the menu.
To save the debug traces select on the debug window the “Capture ” button, give a name and a place
were the traces to be stored

Problem solved
Not applicable

4.6 [HSI][xB] When performing MT120 SWR trough TCIF it is possible that a
MT120 xB board becomes misaligned (red) in TC-NEM's "Rack Overwiev"
with the "Reason: File List Report Failure"

Reference FR
3BKA37FBR269637

Problem description
After activation of a new software when performing MT120 SWR trough TCIF an MT120 xB board to
become misaligned (red) in TC-NEM's "Rack Overwiev" with the "Reason: File List Report Failure".

Corrective action
Possible workarounds:
1. Connect locally to the misaligned (red) MT120 board and activate the new software version, so that it
becomes inline with the software on the other boards.
2. Perform "Reject MT120 Software" from TC-NEM's "Download" window and start the SWR again.
3. Associate the misaligned MT120 board to a dummy BSC and start the SWR again.

Problem solved
Not applicable

ED01 RELEASE01 TC Troubleshooting Guide B11 Release

Site 3BK 29873 MAAA PWZZA 22/27
 4.7 [HSI]Sporadically MT120 board continuously connect/disconnect to TCIF

Reference FR
3BKA37FBR262620

Problem description
Sometimes after performing "Write Conf" or Reset or similar actions that require a MT120 board reset, it
was observed that one board is appearing and disappearing continuously in TC-NEM.

Corrective action
Workaround:
In order to make the MT120 board to be stable in TC-NEM several actions were performed:
- Restart from TC-NEM => no success
- Reset from TC-NEM => no success
All rights reserved. Passing on and copying of this
document, use and communication of its contents

- Unplug/plug of MT120 board from the TC rack => no success

- Exchange the MT120 board with another one => still the same behavior
- Stop/Start of "heartbeat" process => no success
- Takeover => success!

Problem solved
Not applicable

4.8 How to activate the factory software on legacy board

Reference FR
None

Problem description
In case of SW corruption or the package size of the SW is too big than it is necessary to put back the on
the board the factory software.

Corrective action
- Open the Dterm window
- From the terminal menu select "Settings" , then "Itf. Communication Settings"
- Select the following settings:

- Port: COMx
- Data bits : 8 bit
- Parity : none
- Stop bits: 1 bit
- Protocol: none

-Select Baudrate 38400

After the settings done the push the RS232 button from the menu.
-Unplug/Plug the board in slot
- Press in continue S key till than appears on the screen “List of task used by DKOS”
- Press <Ctrl><U> keys
-Write when CMDS>> prompt appears:
flrz 400000 30

ED01 RELEASE01 TC Troubleshooting Guide B11 Release

Site 3BK 29873 MAAA PWZZA 23/27
 If the flash is successfully erase write:

CMDS>> re h

else do again:
-Unplug/Plug the board in slot
- Press in continue S key till than appears on the screen “List of task used by DKOS”
- Press <Ctrl><U> keys
-Write when CMDS>> prompt appears:
flrz 400000 30

Problem solved
Not applicable
All rights reserved. Passing on and copying of this
document, use and communication of its contents

5 TC-NEM Administration

5.1 TC NEM installed on a Windows Vista PC rejects the Local setting although a
local connection is set between TC NEM and TCIF

Reference FR
3BKA37FBR254402

Problem description
On the PC with Windows Vista, during TCIF IP Configuration phase, the TC NEM is opened and the IP
address is introduced and select Local, the settings are rejected with the following error/warning
message: "Please change the IP Address or change to Remote!"

Corrective action
If Remote setting is selected the TC-NEM is connected to the TCIF

Problem solved
Not applicable

5.2 Sometimes one or more processes from TC-NEM remain active in Windows
and need to be close manually from Task Manager, in order to be able to
restart TC-NEM again.

Reference FR
None

Problem description
One or more processes (called for example TC32Container or TC32TCIF) from TC-NEM remain active
in and need to be close manually from Task Manager, in order to be able to restart TC-NEM again, after
the application has encountered a problem and might have suffered a crash.
ED01 RELEASE01 TC Troubleshooting Guide B11 Release
Site 3BK 29873 MAAA PWZZA 24/27
 Corrective action
Close manually from Task Manager the hanging processes.

Problem solved
Not applicable

5.3 The TCIF window is not opened when connect the TC NEM to TCIF

Reference FR
3BKA37FBR254401
All rights reserved. Passing on and copying of this
document, use and communication of its contents

Problem description
On the PC with Windows Vista, during TCIF IP Configuration phase, the TC NEM is opened using the
active TCIF IP. The connection is established but the TCIF window is not opened, so no action can be
performed from TC-NEM.

Corrective action
Before starting the TC NEM, launch the following application files from C:\ProgramFiles\Alcatel-
Lucent\TC-NEM
TC32Controller.exe
TC32OTP.exe
TC32OTPIP.exe
TC32TCIF.exe

Problem solved
Not applicable

5.4 Changing the Management Mode for a MT120 Legacy or MT120 xB board
from HSI to TCIL or TCIL to HSI generates a timeout

Reference FR
None

Problem description
On the serial connection when changing the mode from HSI to TCIL for a MT board the message
“Timeout, request not answered” can be seen in TC-NEM. This message is normal, because the change
in the management mode generates a restart of the board, time in which TC-NEM is not receiving any
information from the board.

ED01 RELEASE01 TC Troubleshooting Guide B11 Release

Site 3BK 29873 MAAA PWZZA 25/27
 Corrective action
After 1 minute disconnect/reconnect TC-NEM to update the view and see all boards that are in TCIL
mode.

Problem solved
Not applicable

5.5 Sometimes, in TC-NEM it is not possible to see any alarm about MT boards
after a request

Reference FR
3BKA37FBR252775
All rights reserved. Passing on and copying of this
document, use and communication of its contents

Problem description
TC-NEM is connected to the TCIF board, through IP.
Several actions were performed on MT boards: reset, restart.
But when triggering enable_fault_indication on one/all boards, nothing happens, no alarm is seen, no
colour in menu window on the MT boards. After a few seconds a message is written in the bottom of the
window: "Timeout, request not answered".
If a command for checking the DSP status is triggered, again no answer.
It seems the MMI process is freezed. But the MMIrelay, oaed and other processes are present and
running on TCIF board..

Corrective action
Close/open again the TC-NEM.

Problem solved
Not applicable

5.6 How to save the TC-NEM log

Reference FR
None

Problem description
Helps the programmer to follow the steps made by the application.

Corrective action
In order to save the TC-NEM logs the following steps have to be performed:
- connect TC-NEM
- in TC-NEM menu select File - Logfile- New- enter the desired name of the log and the place to be
stored
At the end of the test
- in TC-NEM menu select File- Logfile – Close

Problem solved
Not applicable
ED01 RELEASE01 TC Troubleshooting Guide B11 Release
Site 3BK 29873 MAAA PWZZA 26/27
 5.7 Sometimes when performing MT120 replacement with the Wizard provided
by TC-NEM it is possible that the application remains hanging.

Reference FR
3BKA37FBR269891

Problem description
TC-NEM remains hanging when performing MT120 replacement.

Corrective action
To restart TC-NEM, all related processes must be stopped manually from Window's Task Manger
All rights reserved. Passing on and copying of this
document, use and communication of its contents

Processes window.

Problem solved
Not applicable

ED01 RELEASE01 TC Troubleshooting Guide B11 Release

Site 3BK 29873 MAAA PWZZA 27/27