Moveout Upgrade problem

Possible problems meet during the EMS server upgrade or moveout. ¶

1. The received traps display the same host(seems like kind of VIP) on etssvr after change snmp VIP, should display source host. ¶

# abnormal
03:22:16.551869 IP jfk02-wxpd-lb102-vl81-snip.webex.com.35729 > 10.252.56.142.snmptrap:  V2Trap(310)  system.sysUpTime.0=1697599336 S:1.1.4.1.0=E:11435.11.5.1.80 E:11435.11.2.2.1.3.0="10.248.172.178" E:11435.11.2.2.1.2.0="TAS" E:11435.11.2.2.1.10.0=2 E:11435.11.4.2.1.21.1.2.0="GlobalDB-Primary" E:11435.11.4.2.1.21.1.21.0="spGetCustomizedAudioPromptCfg" E:11435.11.4.2.1.21.1.22.0=2561 E:11435.11.2.2.1.6.0="Wed Oct 18 03:22:13 2023" E:11435.1.8.2.1.8.0=5830
03:32:02.386759 IP jfk02-wxpd-lb102-vl81-snip.webex.com.35729 > 10.252.56.142.snmptrap:  V2Trap(310)  system.sysUpTime.0=1697599922 S:1.1.4.1.0=E:11435.11.5.1.80 E:11435.11.2.2.1.3.0="10.248.172.177" E:11435.11.2.2.1.2.0="TAS" E:11435.11.2.2.1.10.0=1 E:11435.11.4.2.1.21.1.2.0="GlobalDB-Primary" E:11435.11.4.2.1.21.1.21.0="spGetCustomizedAudioPromptCfg" E:11435.11.4.2.1.21.1.22.0=2563 E:11435.11.2.2.1.6.0="Wed Oct 18 03:31:59 2023" E:11435.1.8.2.1.8.0=5831

# normal
03:23:33.029730 IP me35taesb104.webex.com.50038 > mtsj1ets003.webex.com.snmptrap:  C="l4z3r" V2Trap(274)  system.sysUpTime.0=2963294269 S:1.1.4.1.0=E:cisco.9.41.2.0.1 E:cisco.9.41.1.2.3.1.2.9342086="VOICE_IEC" E:cisco.9.41.1.2.3.1.3.9342086=4 E:cisco.9.41.1.2.3.1.4.9342086="GW" E:cisco.9.41.1.2.3.1.5.9342086="SIP: Internal Error (Gateway or Interface OOS): IEC=1.1.187.7.14.1 on callID -1 " E:cisco.9.41.1.2.3.1.6.9342086=2963294269
03:23:33.106802 IP me35taesb105.webex.com.54674 > mtsj1ets003.webex.com.snmptrap:  C="l4z3r" V2Trap(274)  system.sysUpTime.0=2781922316 S:1.1.4.1.0=E:cisco.9.41.2.0.1 E:cisco.9.41.1.2.3.1.2.16113982="VOICE_IEC" E:cisco.9.41.1.2.3.1.3.16113982=4 E:cisco.9.41.1.2.3.1.4.16113982="GW" E:cisco.9.41.1.2.3.1.5.16113982="SIP: Internal Error (Gateway or Interface OOS): IEC=1.1.187.7.14.1 on callID -1 " E:cisco.9.41.1.2.3.1.6.16113982=2781922316

Solution: This is because the VIP is not properly configured, ask VIP team member for help, like Jianguo Huang, Jones Zhao ¶

2. The machine in JFK DC cannot conntect to "mail.cisco.com", and "telnet mail.cisco.com 993" timeout, log in file `/var/log/emsd.log`. ¶

Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/opt/emspython/lib/python3.7/imaplib.py", line 1288, in __init__
    IMAP4.__init__(self, host, port)
  File "/opt/emspython/lib/python3.7/imaplib.py", line 198, in __init__
    self.open(host, port)
  File "/opt/emspython/lib/python3.7/imaplib.py", line 1301, in open
    IMAP4.open(self, host, port)
  File "/opt/emspython/lib/python3.7/imaplib.py", line 299, in open
    self.sock = self._create_socket()
  File "/opt/emspython/lib/python3.7/imaplib.py", line 1291, in _create_socket
    sock = IMAP4._create_socket(self)
  File "/opt/emspython/lib/python3.7/imaplib.py", line 289, in _create_socket
    return socket.create_connection((host, self.port))
  File "/opt/emspython/lib/python3.7/socket.py", line 727, in create_connection
    raise err
  File "/opt/emspython/lib/python3.7/socket.py", line 716, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out

Solution: It is a CISCO network problem, not webex network problem, raise a service request(ACL) to CISCO team. eg. RITM2743177 ¶

The Webex PSA for Cisco will be Jason Daniel. The EMS PSA will be Kelly Mockler, we can ask them for help. ¶

3. Old etssvrs still receive traps after change snmp VIP and traps do not swith to new VIP. ¶

Some Xnode machines and devices DNS cache not updated, the IP cache for DNS resolution is still old.

Soultion: Ask Compute Team Own(Eric Yuan, Jacky Chen) for help to restart compute and devices, UCS mgr and Vcenter also belonging to their team(RITM0320212). ¶

The Xnodes can only wait for their next update to restart the service. Xnode enhance JIRA ¶

4. Episvr/imssvr log file has error as below. ¶

An error message appears in the log file: "ImportError: libmysqlclient.so.13" or "ImportError: libmysqlclient.so.18"

Soultion:Due to missing dependencies, need to execute "yum install -y mariadb-connector-c" for install it. ¶

5.If the alarm method is email, and the escalation rule is matched, but the email is not actually sent. ¶

Soultion: ¶

Modify the postfix configuration on the ewb machine. And change the value of relayhost to "mda.webex.com".
[root@mtbt4ewb001 ~]# vi /etc/postfix/main.cf
# Then restart postfix
[root@mtbt4ewb001 ~]# systemctl restart postfix

6. If you can start the service through the python command, but the following error will appear when executing the systemctl command.such as,emsd.service and apihandler.service ¶

Soultion:This situation need to add "/bin/bash" before the script path as below, so that the kernel knows which interpreter to use. ¶

[Service]
Type=forking
User=root
Group=root
PIDFile=/tmp/emsd.pid
ExecStart=/bin/bash /webapp/ecenter/WBXemsd.sh start
ExecStop=/bin/bash /webapp/ecenter/WBXemsd.sh stop
ExecStatus=/bin/bash /webapp/ecenter/WBXemsd.sh status
ExecRestart=/bin/bash /webapp/ecenter/WBXemsd.sh restart
ExecCondrestart=/bin/bash /webapp/ecenter/WBXemsd.sh condrestart
ExecReload=/bin/bash /webapp/ecenter/WBXemsd.sh reload

# Then reload configration and start service again
[root@mthf1ims001-alma8 ~]# systemctl daemon-reload
[root@mthf1ims001-alma8 ~]# systemctl start emsd.service
[root@mthf1ims001-alma8 ~]# systemctl status emsd

alse can move the shell file to /usr/local/bin/ [Service] Type=forking User=root Group=root PIDFile=/tmp/emstrapd.pid ExecStart=/usr/local/bin/WBXemstrapd.sh start ExecStop=/usr/local/bin/WBXemstrapd.sh stop ExecStatus=/usr/local/bin/WBXemstrapd.sh status ExecRestart=/usr/local/bin/WBXemstrapd.sh restart ExecCondrestart=/usr/local/bin/WBXemstrapd.sh condrestart ExecReload=/usr/local/bin/WBXemstrapd.sh reload

7. An error occurs when start redis.(Detail for: https://sqbu-github.cisco.com/Monitoring/docs/blob/moveout-upgrade-problem/docs/ems/knowledge-base/install-redis-cluster.md) ¶

error while loading shared libraries: libssl.so.10: cannot open shared object file: No such file or directory

Soultion:Update the yum source. If it can be executed successfully, then starting redis again will succeed. ¶

[root@mtbt4egr002 ~]# yum update -y
[root@mtbt4egr002 ~]# /opt/redis/bin/redis-server /opt/redis/conf/6379/redis.conf
[root@mtbt4egr002 ~]# ps -ef|grep redis

¶

8. Yum update timeout during start redis server. ¶

Soultion:Manually install "compat-openssl10-1.0.2o-3.el8.x86_64.rpm" package and start redis server again. ¶

# Manually install rpm package and start redis server again.
[root@mtbt4egr002 ~]# rpm -ivh compat-openssl10-1.0.2o-3.el8.x86_64.rpm --force --nodeps
[root@mtbt4egr002 ~]# /opt/redis/bin/redis-server /opt/redis/conf/6379/redis.conf
[root@mtbt4egr002 ~]# ps -ef|grep redis