I don't often write twice in a month, and I certainly don't want to spam your mailbox, but there are a couple of very important issues which have just come to light - good and bad!
Bad news first: VMware has issued a new VMSA (VMware Security Announcement) for a critical vulnerability in the HTML 5 Client. Read more about what this vulnerability is and how to mitigate it in the first section below.
Now the good news: Veeam Backup and Replication V11 is now generally available! Veeam 11 is the answer to many of the requests we (the users) have been making for years. It is very exciting, and you can read more about our adoption plan below.
I've also included some tips to how to handle a full power outage in a VMware vSphere environment. WIth the recent climate and utility issues, some of our clients have had to face this very real possibility and I wanted to provide the lowdown on what to do and expect.
Thanks for taking a minute to read our second email this month and I hope that we are providing useful information. If you have any questions, comments, or input, please contact me personally. If you just don't want to receive the emails, just click "unsubscribe" below.
The vSphere Client (HTML5) contains a remote code execution vulnerability in a vCenter Server plugin. VMware has evaluated the severity of this issue to be in the Critical severity range with a maximum CVSSv3 base score of 9.8.
What products are covered:
VMware vCenter Server (VCSA)
VMware Cloud Foundation
VMsources estimated impact:
This is a very serious issue and all users should patch immediately. Nonetheless, if your instances of affected products are on separate (firewalled) management networks, the impact is mitigated to those users who can reach the products on port 443.
How to mitigate:
Take a backup of your affected product
Connect to the VAMI interface of your affected product (https://<product-url>:5480
Log-in with the username: root and your root password
Locate the updates section
Update your affected product
Veeam 11 General Availability
This is a very exciting moment in the history of Veeam, as it introuduces Continuous Data Protection (CDP) and Immutable Backups.
Veeam Continuous Data Protection
Veeam CDP provides continuous data transfer from a source VM to a target VM, with little to no Replication Point Objective (RPO)! This is accomplished with a VMware-certified I/O filter installed on each ESXi in the Cluster constantly sending changes as they occur.
This is the technology we have all been waiting for, except it comes at some cost. Veeam CDP has the following requirements:
Production-grade SAN storage for both source and target VM
Greater bandwidth consumption than standard replication
Higher Compute overhead per item protected on both source and target vSphere Cluster
That being said, Veeam CDP has the following advantages:
No VMware snapshots or snapshot stun
No vendor or hardware dependence (use any enterprise storage or SAN)
No RPO or schedule to monitor as it is continuous.
Veeam Immutable Backups
One of the biggest challenges of recent years has been Ransomware and/or malicious employees who intentionally encrypt or delete backup data, making recovery impossible after an event.
While S3 Cloud storage can be marked immutable, and WORM Tapes have been available for years, recovery could be painfully slow, and tape archives are expensive to maintain at any scale.
Now Veeam supports Immutable backups on Linux repositories. Through the use of certificate-only login and the immutable property of Linux filesystems, Veeam has designed a true WORM equivalent with the advantages of a random-access repository!
To create your own Immutable Repository for Veeam, you will need to build a physical Linux server (recommended) or VM with adequate XFS storage. VMsources and Veeam recommend Ubuntu 20.04 LTS because of its known lifecycle and long pedigree (and also because of what RHEL did to CentOS).
What's the timetable for adopting Veeam 11?
At VMsources, we are going to begin testing right away and let everyone know what our experience has been like, and what the problems are "out in the wild".
Often on the release of major new feature sets such as Veeam 11 introduces, major issues are discovered weeks or even months after General Availability (GA). VMsources feels that these are groundbreaking and important features and we intend to adopt Veeam 11 as soon as our Change Management policy allows.
What that means is that we'll test Veeam 11 for at least 30 days ourselves before we certify it for production use. During that time, not only will we be testing Veeam 11 ourselves, but we'll be monitoring the Veeam forums and news for incidents and possible data loss.
We recommend waiting at least one month to upgrade to Veeam 11. If all goes well, we plan on upgrading our own hosted Veeam instances to version 11 on April 3 & 4 2021.
Power Outages for VMware vSphere
Before the power goes out
If you know that you are going to lose power (as is the case when mains power has failed and you are running on UPS power) it's always a good idea to begin to shut your VMs down, beginning with transactional VMs such as databases.
Remember, if you shut it down you are also disabling HA for that VM andyou will need to turn it back on after the power is restored.
When power is restored to the system
Thee boot order should always be:
SAN and storage systems
Then allow 15 to 20 minutes, because after a full shutdown a "boot storm" occurs causing high storage latency.
Once the ESXi hosts have booted
HA Will automatically restart any VMs that were powered on at the time ESXi host power was lost. In the event that all VMs were running when the power outage occurred, you can expect HA will automatically start all VMs.
As we mentioned earlier, if you had the opportunity to shut down transactional VMs prior to a power outage, these VMs will need to be manually restarted by an authorized administrator.
10 to 15 minutes after all of the ESXi hosts have finished booting, You should be able to login to vCenter/VCSA. Browse to:
There may be some/many objects marked as "orphan" and you may see the same VM on multiple hosts, these are relics of HA action in the absence of vCenter/VCSA - DO NOT attempt to clean up! vCenter/VCSA will take care of removing duplicates and orphans as soon as it is running.
Once you have located the vCenter/VCSA, right-click to manually power it on and wait for it to fully boot before taking any further actions, usually 3-5 minutes.
Once the vCenter/VCSA is fully booted you will be able to manually re-power any transactional VMs which you havd the opportunity to shut down prior to the power outage
Don't forget to clear any lingering alarms that were caused as a result of the power outage such as network uplink redundancy lost.