Sep 28, 2018

Behind the Scenes : vRealize Automation Upgrade to 7.5

Arun Nukula
Sep 28, 2018
6 min read

Rated NaN out of 5 stars.

We monitor /opt/vmware/var/log/vami/updatecli.log and upgrade-iaas.log during upgrade of vRealize Automation , it might be any version for that matter.

Was working on an upgrade to vRA 7.5 today and decided to look behind the scenes on what exactly does it log under above mentioned logs.

So let’s start our deep-dive into them.

Upgrade Starts
Starts Running pre-install scripts
Pre-Install is set to “IN PRORGESS”
Then it disables database automatic failover ( SYNCHRONOUS TO ASYNCHRONOUS )
Run’s abort-on-replica script , at this point it checks if the MASTER version is lesser that it’s REPLICA version
Then creates a copy the upgrade repository to another location so it can be used in postupgrade scripts. This happens only on the MASTER node
The following script checks if the hardware resources are enough for the newly installable vRA version
Checks vRA Replica hosts availability for source versions >= 7.1 , if Replica hosts are not available then it would throw an exception to fix them before the upgrade
If check’s for any vRO duplicates and if none if proceeds with the next step. If it finds any then it would delete them
At this point it upgrades MANAGEMENT AGENTS on all IAAS nodes
Post Management Agent’s upgrade , pre-requisite checks on IAAS node’s start. This is the point where it checks if we have included IAAS upgrade or we have excluded it and upgrading it manually. If it finds /tmp/disable-iaas-upgrade , then it’s going to disable or skip all pre-req steps on IAAS nodes
Identifies or Generates cluster node id
Checks if vRB service is registered. If REGISTERED then it’s going to let us know if that version of vRB is compatible with the version of vRA we are upgrading to. Else , it would let us know that’s there is a compatibility issue but it would not stop the upgrade
Validates if there are blueprints in the system that cannot be migrated automatically. If such blueprints are found the upgrade will be blocked
Maps LB to localhost
Kills all Java Processes. It executes vcac-server , vco-server and vco-configurator stop commands
Cleans any temporary files under /var/lib/vco/*
Formats /dev/sdd and moves database to it. /opt and /var are moved to the older db partition.
Applies few fixes w.r.t extending partition
Checks if /dev/sda is 50 GB and resize partitions according to the new existing space
Kills Health Broker service and monitor before upgrading
Stops IaaS services in the order: Agent, DemWorker, DemOrchestrator, ManagerService
Stops vRA services on all the Replica hosts for source versions >= 7.1
Vacuums database only from the primary VA
This script is used to dump vco database and import it to vcac. It’s only for older versions
This script is used to fix location where packages are downloaded because in 6.2.x and earlier 7.x ,versions there is not enough space in the root partition. If there is not enough space , then it’s going to create one and move the content into it
Uninstalls the artifactory rpms from the appliance
This script is used to dump PostgreSQL databases in case of major upgrade. Checks if dump/restore is needed at all and exit if major versions are the same
Executes a script which is a workaround for an issue when upgrading vmStudio from versions prior to 3.0.0.0. If the current vmStudio version is prior to 3.0.0.0, then forces deletetion of the vmware-studio-vami-cimom package
Removes persistent net rules
Artifactory uninstall fix starts
Saves the JRE cacerts file as some of them are imported by horizon
Removes any resource bundles
Stop psql-manager service at the beginning of post update operations, as it might not be able to connect to the database, thus it will not be able to see that it is in async mode and will try to perform a reset. It will be started again at the end of the post update
After PostgreSQL database is processed (still in preupdate step), this script will check if there is already exported database (which means major upgrade) then ,Will ensure PostgreSQL is stopped and delete existing data and server directory
Prepares various services to stop
It mark’s Pre-Install tasks as complete
Now it start’s running installation tests
Start’s package installation
Now that package installation is complete , it would start running post installation scripts
Preserves DB settings , copies /etc/vcac/server.xml to /tmp
Performs rpm status checks
Stop psql-manager service at the beginning of post update operations, as it might not be able to connect to the database, thus it will not be able to see that it is in async mode and will try to perform a reset. It will be started again at the end of the post update.
Creates a file /tmp/vra-upgrade-on.txt
Checks if there are any external databases to be merged
Ensures that all local users will not expire
Ensure the keys are not already in the file /etc/sysctl.conf
This script is used to fix location where packages are downloaded because in 6.2.x and earlier 7.x ,versions there is not enough space in the root partition.The script will not run (see package-pool.inc) if the /opt is symlink, which means it is moved to /storage/ext
Stop PostgreSQL server and ensure data directory is on the correct partition, checks or set’s the MASTER in Database
Checks recovery.conf and updates other config files on postgres
This script is used to restore PostgreSQL databases previously exported (in preupdate step) in case of major upgrade
Starts sshd after the update if it is enabled
Prepares required services
Initialize users and generate encrypted pwd for administrator foe XENON. And it initializes XENON
Initialize users and generate pswds for vrhb
Performs cleanup of sandbox dir. In upgrade from previous system than 7.5 it’s mandatory. Upgrade from 7.5 and later, sandbox folder will contains only the static UI files that are regenerated from the host
Calls firstboot scripts for postgres clustering
Only if D is external and local database is replica ( in the case of 6.3 with external lb ) , the db replica state will be cleared ,else it would exit
Patches the rabbitmq scripts for the sed options
Removing persistent net rules
Prepares required vcac services
Updates database and then creates tables used by vcac-config
Removes truststore
Adds vCO system properties in vmo.properties and enables vra mode in ControlCenter
Reencrypts keystore password
Changing the hzn master keystore password if it’s been set to a default one
Applies fix for issue with value: none for property certificate.store.ssl.rabbit.alias
This script will disable particular PIDs from hardening scripts being invoked after upgrade
Replaces the update URL
Reconfigures vco
Add additional lighttpd configurations directory
Just logs version
Removes old log file that is not used anymore - the same messages are in /var/log/vmware/vcac/vcac-config.log
Executes set guest to export vami variables (they were not exported in the old versions)
Deploys all vRA services to tomcat
Remove the orig file created by the studio build process /etc/init.d/rc
Updates the java timezone data
Pinning telemetry log collection runs only on master. Disables them on Replica
Setting up log symlinks for /storage/log/
Set’s coredumps under /storage/core
Fixes sshd config
Edits sysctl.conf and pushes configurations into it
Additional lighttpd configuration goes to the /opt/vmware/etc/lighttpd/conf.d directory - removes old config if there is any
Configures allowed services under /etc/hosts.allow
dodscript.sh makes a symlink for the /etc/issue file - does not overwrite it instead writes to /etc/issue.ORIG
Set’s and customizes grub timeout
Fixes one of the vami_ovf bugs
Fixes for another bug
Fixes for another bug
Disable screen blanking on tty1
Links vmware-rpctool where vami expects it
Patches vami_set_hostname
Another bug fix
Add user root to wheel group otherwise it cannot login with SSH because of hardening scripts
Fixes tcserver startup flles
Disable chroot for ntpd
Adds lighttpd headers
Patches VAMI css
Patches vami-deploy.xm
Executes haproxy fix
Enables haproxy
Deletes postgres export directory
Deletes legacy services
Copies openscap branding
Removes multiple tomcat servers if existed
Applies default ciphers for SFCB server
Starts psql manager
Marks to trigger automated IAAS upgrade after node’s reboot
Checks RabbitMQ node health. Starts upgrade on the replica nodes or VA’s. Now you would see that there would be no logging for a long time on the MASTER until the replica’s are upgraded
Deletes /tmp/disable-iaas-upgrade , if it was created before ( incase of manual iaas upgrade)
- for script in ’”${bootstrap_dir}”/*’
Ciphers updated
Flag set for vami_setnetwork
Applies workaround for kernel hanging on non-available cdrom device
Migrates custom groups if any
Posts iaas upgrade messages

“After all appliances are upgraded, ssh to the master appliance and go to /usr/lib/vcac/tools/upgrade and execute the ”./upgrade” ”

“Wait for step *Post-install* to complete and then reboot the master appliance. After it is rebooted, IaaS nodes upgrade will commence automatically.Progress will be displayed on *this page* on the master appliance”

All Appliances are now upgraded
Completes Post Install scripts and finishes reconfiguration
Finalizes Installation
Complete Upgrade on Appliance successfully on MASTER and REPLICA’s
Now that Virtual Appliance Upgrade is complete , we will reboot the MASTER node. Once we reboot MASTER while it’s starting up at a point where it brings up network interfaces and start’s application services , it would initiate a automatic reboot on other REPLICA nodesOnce all Services on the MASTER node are REGISTERED , IAAS Upgrade would kick in
IAAS Upgrade starts, upgrades components on my two of IAAS nodes
Disables Maintenance mode on second IAAS node
Upgrades DEM’s
Upgrades Proxy Agents
Enables Manager Service Automatic Failover mode
Finally completes the upgrade and restores Postgres Replication mode back to SYNCHRONOUS mode

That’s Curtains for your vRA Upgrade…

Download complete updatecli.log here