Having Multi-Tenancy enabled in vRO causes vRA Upgrade failures during the pre-update phase
Having Multi-Tenancy enabled in vRO causes vRA Upgrade failures during the pre-update phase
-
-
Aug 3, 2020
-
4 min read
Rated NaN out of 5 stars.
Failures are seen during vRA upgrade through vRSLCM 2.1during preupdate phase of vRA appliance upgrade
Upgrade was being performed from vRA 7.4 to 7.6
vRLCM was on version 2.1 Patch 2
/var/log/bootstrap/preupdate.log
Running a check on replicas:
Executing a script on multiple cluster nodes...
2020-05-28 05:34:18+00:00 /etc/bootstrap/preupdate.d/00-00-01-va-resources-check done, succeeded.
2020-05-28 05:34:18+00:00 /etc/bootstrap/preupdate.d/00-00-02-check-replica-availability starting...
Executing a script on multiple cluster nodes...
2020-05-28 05:34:26+00:00 /etc/bootstrap/preupdate.d/00-00-02-check-replica-availability done, succeeded.
2020-05-28 05:34:26+00:00 /etc/bootstrap/preupdate.d/00-00-02-check-vro-duplicates starting...
<!--: spam
Content-Type: text/html
<body bgcolor="#f0f0f8"><font color="#f0f0f8" size="-5"> -->
<body bgcolor="#f0f0f8"><font color="#f0f0f8" size="-5"> --> -->
</font> </font> </font> </script> </object> </blockquote> </pre>
</table> </table> </table> </table> </table> </font> </font> </font><body bgcolor="#f0f0f8">
<table width="100%" cellspacing=0 cellpadding=2 border=0 summary="heading">
<tr bgcolor="#6622aa">
<td valign=bottom> <br>
<font color="#ffffff" face="helvetica, arial"> <br><big><big><strong><type 'exceptions.TypeError'></strong></big></big></font></td
<td align=right valign=bottom
<font color="#ffffff" face="helvetica, arial">Python 2.7.14: /usr/bin/python<br>Thu May 28 05:34:28 2020</font></td></tr></table>
<p>A problem occurred in a Python script. Here is the sequence of
function calls leading up to the error, in the order they occurred.</p>
<table width="100%" cellspacing=0 cellpadding=0 border=0>
<tr><td bgcolor="#d8bbff"><big> </big><a href="file:///etc/bootstrap/preupdate.d/00-00-02-check-vro-duplicates">/etc/bootstrap/preupdate.d/00-00-02-check-vro-duplicates</a> in <strong><module></strong>()</td></tr>
<tr><td><font color="#909090"><tt> <small> 51</small> for dup in duplicates_check_result:<br>
</tt></font></td></tr>
<tr><td><font color="#909090"><tt> <small> 52</small> dup_line = "{} items in table {}: {} {}".format(str(dup.get('?
column?')), dup.get('type'),<br>
</tt></font></td></tr>
<tr><td bgcolor="#ffccee"><tt>=><small> 53</small>  
; '' if len(dup.get('categoryid')) == 0 
;else "ID=" + dup.get('categoryid'),<br>
</tt></td></tr>
<tr><td><font color="#909090"><tt> <small> 54</small>
'' if len(dup.get('name')) == 0
else "NAME=" + dup.get('name'))<br>
</tt></font></td></tr>
<tr><td><font color="#909090"><tt> <small> 55</small> msg_html += "<p style='margin-left:20px'>- {}</p>".format(dup_line)<br
</tt></font></td></tr>
<tr><td><small><font color="#909090"><em>builtin</em> <strong>len</strong> = <built-in function len>, <strong>dup</strong> = {'?column?': 1L, 'type': 'vmo_configelementcategory', 'categoryid': None, 'name': 'wbg_SQL'}, du
p.<strong>get</strong> = <built-in method get of RealDictRow object></font></small></td></tr></table><p><strong><type 'exceptions.TypeError'></strong>: object of type 'NoneType' has no len()
<br><tt><small> </small> </tt>args =
("object of type 'NoneType' has no len()",)
<br><tt><small> </small> </tt>message =
"object of type 'NoneType' has no len()"
<!-- The above is a description of an error in a Python program, formatted
for a Web browser because the 'cgitb' module was enabled. In case you
are not reading this in a Web browser, here is the original traceback:
Traceback (most recent call last):/var/log/messages
2020-05-28T05:34:29.194212+00:00 sevenfour kernel: [ 5275.524647] martian source 10.63.249.208 from 0.0.0.0, on dev eth0
2020-05-28T05:34:29.194252+00:00 sevenfour kernel: [ 5275.524657] ll header: ff:ff:ff:ff:ff:ff:00:00:00:00:7b:00:08:00
2020-05-28T05:34:31.588050+00:00 sevenfour vami /usr/lib/vcac/agent/va-agent.py[6316]: info .................................................................................................................................................
.............................................................................................................................................................................................................................................
...................................................................
2020-05-28T05:34:31.588183+00:00 sevenfour vami /usr/lib/vcac/agent/va-agent.py[6316]: info Failure: Error while running pre-install scripts and post-install scripts
2020-05-28T05:34:31.588194+00:00 sevenfour vami /usr/lib/vcac/agent/va-agent.py[6316]: info
2020-05-28T05:34:31.603338+00:00 sevenfour vami /usr/lib/vcac/agent/va-agent.py[6316]: info <p>VA-check: finished</p>
2020-05-28T05:34:31.603561+00:00 sevenfour vami /usr/lib/vcac/agent/va-agent.py[6316]: info <p>Pre-install: failed (code p-1)</p>
2020-05-28T05:34:31.603870+00:00 sevenfour vami /usr/lib/vcac/agent/va-agent.py[6316]: info <p>RPM Status 1: <span style='color: red'>Pre install script failed, package test and installation skipped</span>.</p>
2020-05-28T05:34:31.604050+00:00 sevenfour vami /usr/lib/vcac/agent/va-agent.py[6316]: info <p>Post-install: failed</p>
2020-05-28T05:34:31.604227+00:00 sevenfour vami /usr/lib/vcac/agent/va-agent.py[6316]: info <p>Update failed (code 1-1). Check logs in /opt/vmware/var/log/vami or retry update later.</p>
2020-05-28T05:34:31.604354+00:00 sevenfour vami /usr/lib/vcac/agent/va-agent.py[6316]: info <br />
2020-05-28T05:34:31.607027+00:00 sevenfour vami /usr/lib/vcac/agent/va-agent.py[6316]: info Finished..
2020-05-28T05:34:31.607088+00:00 sevenfour vami /usr/lib/vcac/agent/va-agent.py[6316]: info Executing shell command...
2020-05-28T05:34:35.725031+00:00 sevenfour vami /usr/lib/vcac/agent/va-agent.py[6316]: info Calling cluster-command-get-next
2020-05-28T05:34:35.725081+00:00 sevenfour vami /usr/lib/vcac/agent/va-agent.py[6316]: info Executing shell command...Executing KB 54987 shows that there is no database configuration update needed. Which means there are actually no duplicates
sevenfour:/tmp # /usr/lib/vco-cli/bin/vro-configure.sh db-validate
Orchestrator's root folder: /var/lib/vco
Orchestrator Configuration Tool. Version: 7.4.0.8989539 Build: 8989539
ValidateDbOptions{dbUsername='vcac', dbPassword='****', jdbcUrl='jdbc:postgresql://127.0.0.1:5433/vcac?sslmode=verify-ca&sslrootcert=/var/vmware/vpostgres/current/.postgresql/root.crt'}
The command does not need a database configuration update.Even then the precheck fails with the exception stating there are duplicates. This is because of Multi-Tenancy enabled in vRO
The duplicates in vRO are coming from the fact that they are using a multi-tenanted environment and vro duplicates script is not designed for that.
Remediation Plan
This is how you remediate this problem so that a successful upgrade can be performed
Step:1
Extract the attached archive 00-00-00-00-remove-check-vro-duplicates.zip into the VA you are going to upgrade In the folder: /etc/bootstrap/preupdate.d
00-00-00-00-remove-check-vro-duplicates
Download • 785B
Step:2
Execute below command to change permissions
chmod 744 /etc/bootstrap/preupdate.d/00-00-00-00-remove-check-vro-duplicatesThen start the upgrade either through vRLCM or through VAMI.
This should help in progressing upgrade further without any failures.
What is this additional script doing?
This script 00-00-00-00-remove-check-vro-duplicates deletes 00-00-02-check-vro-duplicates
when triggered
#!/bin/sh
rm -f /etc/bootstrap/preupdate.d/00-00-02-check-vro-duplicatesIdeally /etc/bootstrap/preupdate.d/* contains all the scripts which has to run during preupdate phase of vRA upgrade
[master] sevenvra:/etc/bootstrap/preupdate.d # ls -l
total 160
-rwxr--r-- 1 root root 2284 Mar 28 2019 00-00-00-prereq-checks
-rwxr--r-- 1 root root 781 Mar 28 2019 00-00-00-wizard-completed
-rwxr--r-- 1 root root 655 Mar 28 2019 00-00-00-z-disable-db-failover
-rwxr--r-- 1 root root 1883 Mar 28 2019 00-00-01-abort-on-replica
-rwxr--r-- 1 root root 853 Mar 28 2019 00-00-01-check-replica-availability
-rwxr--r-- 1 root root 918 Mar 28 2019 00-00-01-copy-repository
-rwxr--r-- 1 root root 1377 Mar 28 2019 00-00-01-va-resources-check
-rwxr--r-- 1 root root 3685 Mar 28 2019 00-00-02-check-vro-duplicates
-rwxr--r-- 1 root root 2598 Mar 28 2019 00-00-03-upgrade-management-agents
-rwxr--r-- 1 root root 12433 Mar 28 2019 00-00-04-prereq-iaas
-rwxr--r-- 1 root root 724 Mar 28 2019 00-02-generate-vra-cluster-node-id
-rwxr--r-- 1 root root 1815 Mar 28 2019 00-10-vrb-version-check.sh
-rwxr--r-- 1 root root 1563 Mar 28 2019 00-20-check-blueprints
-rwxr--r-- 1 root root 600 Mar 28 2019 00-20-map-lb-to-localhost
-rwxr--r-- 1 root root 149 Mar 28 2019 00-50-kill-all-java-processes
-rwxr--r-- 1 root root 472 Mar 28 2019 00-60-clean-resources
-rwxr--r-- 1 root root 1685 Mar 28 2019 00-70-va-resources-disk-resize
-rwxr--r-- 1 root root 298 Mar 28 2019 00-71-PR-1590946-ext-partition-fix
-rwxr--r-- 1 root root 2723 Mar 28 2019 00-71-va-root-disk-resize
-rwxr--r-- 1 root root 352 Mar 28 2019 05-stop-vrhb-service
-rwxr--r-- 1 root root 1488 Mar 28 2019 06-stop-iaas-services
-rwxr--r-- 1 root root 3347 Mar 28 2019 06-stop-vra-replica-services
-rwxr--r-- 1 root root 698 Mar 28 2019 07-vacuum-db
-rwxr--r-- 1 root root 2114 Mar 28 2019 08-vco-database
-rwxr--r-- 1 root root 1000 Mar 28 2019 09-60-package-pool
-rwxr--r-- 1 root root 819 Mar 28 2019 09-70-remove-artifactory
-rwx------ 1 root root 5152 Mar 21 2019 10-postgres
-rwx------ 1 root root 1297 Mar 21 2019 10-prepare-vmstudio-update
-rwxr--r-- 1 root root 1547 Mar 28 2019 14-remove-persistent-net-rules
-rwxr--r-- 1 root root 529 Mar 28 2019 20-artifactory-uninstall-fix
-rwxr--r-- 1 root root 253 Mar 28 2019 80-horizon-certs
-rwxr--r-- 1 root root 57 Mar 28 2019 90-remove-resource-bundles
-rwxr--r-- 1 root root 750 Mar 28 2019 98-psql-manager-stop
-rwx------ 1 root root 1291 Mar 21 2019 99-backup-config-files
-rwx------ 1 root root 602 Mar 21 2019 99-postgres
-rwx------ 1 root root 555 Mar 21 2019 99-prepare-servicesSo when an upgrade is executed after vRA rpm’s are copied these scripts are downloaded under preupdate/ path
They are executed sequentially
All we are trying to do is a workaround for an upgrade not to check vro duplicates while running an upgrade resulting in a failure. Hence we are deleting 00-00-02-check-vro-duplicates by inducing another script which will be executed before anything else is executed. As we named it as 00-00-00-00-remove-check-vro-duplicates