vSphere Replication and Traffic Separation

Reading Time 4 Minutes

Recently I had more to do with vSphere Replication and Site Recovery Manager (more specifically with versions 8.6) and would like to share my experiences with Traffic Separation for vSphere Replication with you.

What is Traffic Separation?

Traffic separation is the possibility of splitting network traffic between different networks/port groups/VLANs and thereby possibly achieving an increase in security and performance.

So why separate?

A vSphere Replication Appliance comes with a (virtual) network adapter by default, and this is used for all network communication types.
These are:

  • Management Traffic between vSphere Replication Management Server and vSphere Replication Server, between vCenter and vSphere Replication Management Server
  • Replication traffic from source ESXi hosts to the target vSphere Replication Server.
  • NFC (Network File Copy) replication traffic from the target vSphere Replication Server to the target ESXi datastores.

This VMware KB article also explains the process very well.

Traffic Flow vSphere Replication, Blue = Management Traffic, Orange = Replication Traffic, Green = Replication NFC Traffic

For a better understanding, follow the network traffic in the diagram.
You can see how the data flows from the source ESXI hosts on the left to the target replication appliance (vRMS, orange line) and then from the target vRMS to the target ESXi hosts (green line). The blue lines are the management traffic.
Note: For simplicity, the graph only shows the path in one direction. For a failback and replication in the opposite direction, it would be the other way around.

As already mentioned, all of these types of traffic use the same virtual network adapter of the appliance in the standard configuration, and therefore the same port group and the same VLAN.
Now it may well be that you don’t want to do this, so accordingly: Traffic separation!

Traffic Separation – here’s how

Prerequisites for the following steps are:

  • The already deployed vSphere Replication Appliances in source and target vCenter
  • Port groups on the virtual switches are ready

I have made it easy for myself with regard to traffic separation, as I only want to separate the management traffic from the entire replication traffic.
Accordingly, I only need one replication port group (here alex-replication) in addition to the management port group (in my case alex-srm).
In case you also want to separate the NFC replication traffic, provide another portgroup and add another network card at the part with the network card. #easy!

That’s what it looks like when the appliance is installed and before we start with the adjustments:

vSphere Replication VAMI Networking after Installation

Power off the VM again and add a Network Adapter in the Settings:

Add new Network Adapter

Connect the new adapter to the dedicated replication port group:

Connect new Network Adapter to the Replication Portgroup

Then power on the VM again, after which it looks like this in the VAMI, a new interface “eth1” has appeared:

Check if eth1 shows up in vReplication VAMI

Click “Edit” and then make the settings for “eth1”:

Configure eth1, but with the Gateway of eth0!

The gateway is required, but beware: the gateway that is set here applies “globally”, i.e. for the whole VM and overwrites the gateway of eth0.
For this reason I have entered the same gateway as for eth0.
If the source and target networks are located in different networks, static routes must be used anyway; these are set exclusively via SSH/CLI.

Check again after “Save” and… well, “UP” is good, but what’s going on? Where has my network configuration gone?

eth1 up, but where is the rest of the information?

Don’t panic, you can see the new IP in the VM Summary in vCenter…

All IP addresses visible in the vCenter VM Summary

…or via CLI:

Or check the network config via CLI

Anyway, don’t let this stop you for the time being and continue and register the appliance in the respective vCenters. Set the IP address of eth1 as the “Storage Traffic IP”:

Register the Appliance with the vCenter Part 1
Register the Appliance with the vCenter Part 2

In the Appliance Management interface, the appliance now states in the Summary that the Storage Traffic IP unfortunately does not match any NIC IP. 🤔

Storage Traffic IP does not match?

Okay?! Again, don’t let it bother you for the time being. Let’s move on!

I created a new VMkernel for each of the source and target ESXi hosts and activated the services “vSphere Replication” and “vSphere Replication NFC” (in case anyone has ever wondered what these checkmarks are for, here is the answer 😎)

Why activate both?
Summarised briefly once again:
“Replication” is from source ESXi to target appliance.
“Replication NFC” is from the target appliance to the target ESXi.
If I activate both on the VMkernel, I can then also simply replicate in the opposite direction 👍.

In our lab, I had the luxury of being able to use an NSX segment and have all hosts on the same subnet. This saved me from having to use static routes in the appliances.
(In case someone has separate networks, here is the section from VMware’s documentation for static routing.)

New VMkernel with Replication and Replication NFC enabled

Traffic Separation – the Test

After pairing the two replication appliances via vCenter, I created a replication for a VM to check which interface was actually being used.

I used the following command to display the network statistics of the appliance while replication was active:

watch -n1 --differences ip -s link show up

It was clearly visible that the traffic runs via eth1:

“ip -s link show up” Command, clearly to see more throughput on eth1

In addition, I checked the network of the target ESXi host via esxtop. Here, too, one could see that the new VMkernel vmk2 and the associated vmnic3 were busy.

esxtop Command, the Replication VMkernel is pretty busy.

Conclusion

Apart from the poor implementation of the feature in the Appliance GUI and the impression that it is rather a hidden feature (at least currently with vSphere Replication 8.6), I see Traffic Separation as quite appropriate. Different networks for different applications/functions are simply part of a clean design, enhance security and performance. So why not for replication data as well.

🐵

Replacing the vCenter Machine Certificate …and don’t forget the VxRail Manager!

Reading Time 3 Minutes

The topic of certificates seems to be haunting me at the moment.
Anyway, I want to briefly show here how easy it is nowadays to replace the SSL certificate of the vCenter with an Enterprise CA-signed one.

If you look at the KB article from VMware (Replacing a vSphere 6.x /7.x Machine SSL certificate with a Custom Certificate Authority Signed Certificate), the “certificate-manager” is still quoted here on the command line.

Create CSR in vSphere Client

But it is also very easy via the vSphere Client. In my case, there are a couple of VxRail clusters connected to this vCenter, here you also have to do something in the VxRail Manager (in this case still via CLI), but it’s also easy, see below.
To the Demo!

Login to vCenter and go to Administration >> Certificate Management, then __MACHINE_CERT >> Actions >> Generate Certificate Signing Request:

Generate Certificate Signing Request (CSR)

Put in all info for the Signing Request:

Enter Infos for the CSR

…and either copy or download the CSR:

Copy or Download the CSR

Well, that was easy!
The something.csr file is now required by the certificate admin(s), or, if you do this with the Windows CA, there may also be a self-service portal, but this is not the topic here (but it could work as in this blog post).

The CSR has been submitted, what’s next?

After creating a certificate from the CSR, we should now have a CER file and a certificate chain file, both available from our CA.

Back to vSphere Client, this time Administration >> Certificate Management, then __MACHINE_CERT >> Actions >> Import and Replace Certificate:

Import and Replace Certificate

Then “Replace with external CA certificate where CSR is generated from vCenter Server”:

Replace where CSR was generated on vCenter

Hadn’t mentioned it yet: the advantage of a CSR from the vCenter is that the private key is already “embedded”. With a CSR that was not created on the vCenter, it is missing. You would then have to get it additionally and select option 3.

Anyway, with option 2, it then goes on like this:
Select a new CER file from the CA (Machine SSL Certificate) and select a chain certificate file from the CA (Chain of trusted root certificates) and click Replace.
Attention: now the vCenter services restart! This will not take long, but should probably not be done in productive environments without prior consultation.

Add CER file and Chain-cert

If you are impatient (F5!!! F5!!! F5!!!) and can already log in again, the certificate overview may look a bit strange and also spit out a few errors. Relax and give it a few minutes!

And that’s it, the vCenter now has a “Trusted Certificate” from its own CA and the browser is no longer annoying with security warnings.

The VxRail Manager also wants to get back on!

If there are applications that talk to the vCenter ( this can happen), such as vRealize Log Insight, vRealize Automation, vRealize Operations, Skyline Advisor, VxRail Manager,…. then they do not yet know about their luck and should be adjusted (by re-authentication or similar).

IIn my case, there are a few VxRail Managers here that now have no “trust” with the vCenter, but this can be fixed with a Python script (available from Dell in the support article VxRail: How to manually import vCenter SSL certificate on VxRail Manager). The script is updated from time to time, so always check back first!
So:

  • Load the script onto the VxRail Manager (e.g. via WinSCP).
  • log in as “mystic” via SSH
  • “su” to user “root”
  • execute script: python cert_util.py
VxRail Manager cert_util.py

As you can see, the script loads the new CA certificate from vCenter, replaces the existing ones with it and then restarts a few services. Done.

Easy!

In my experience, the use of self-signed certificates is still a rarity, but the effort is small. The CSR is generated in no time and the certificates are quickly exchanged in the vSphere Client.

vSphere+ – Installation and First Impressions

Reading Time 4 Minutes

Everyone has been talking about vSphere+ for a few weeks now, so I wanted to get a (technical) impression of what’s behind all the marketing-heavy blog articles and announcements.

A short diagram in advance so that it is clear what we are dealing with:

Traditional vSphere Environment converted to vSphere+ Subscription, graphic courtesy of VMware

In a classic vSphere environment, there are one to many vCenters that need to be managed. With vSphere+, a vCenter Cloud Gateway is introduced which acts as a relay between the VMware Cloud and the on-premises vCenters. This allows services from the VMware Cloud to be used with the on-premises datacenter. Sounds pretty easy!

Fortunately, VMware offers a 15-day trial in the VMware Cloud. Yeah, let’s roll! 🚀

Logged on to https://vmc.vmware.com/ with my Customerconnect Portal login details and started the vSphere+ trial:

VMware Cloud Portal
The Trial is running!

Moving on to VMware’s Customer Connect Portal, here you can download the “VMware vCenter Cloud Gateway for vSphere+”.

Logged in under Products and Accounts >> Products >> All Products it can be quickly found:

Customer Connect Portal – All Products
Customer Connect Portal – All Products – Finding the Download
Download the Product

After the download, I open the ISO file on my client computer, in my case Windows, the image is mounted in a “virtual drive”. I navigate through ui-installer >> win32 and start the installer.exe.

Mount ISO File
Open ui-installer Folder
Open appropriate OS folder
Execute installer.exe

The graphical installer window opens.

The Cloud Gateway Appliance Installer

Up to this point, the installation of the appliance was similar to the installation of a vCenter. The following steps are graphically different, but there is not much to consider here, so here is a brief screenshot shower:

It’s just the Deploy Button to hit…
Accept the license agreement

Select a vCenter (or ESXi host) as the installation target for the appliance and provide credentials:

Choose Deployment Location
Accept SSL certificate

Select folder and cluster for installation:

Chose Folder…
…and Cluster/Host for Deployment

Define VM Name incl. root password:

VM name and root password

Select Datastore and Disk Mode:

Chose Datastore and Disk Mode

Then the network settings…

…here I wanted to make it easy for myself and selected an NSX-T segment with a local DHCP server. Unfortunately, the pre-checks for the VMware Cloud connection later showed that the DNS resolution of the appliance did not work (somehow this was logical, how should the Microsoft DNS in our lab get a clue about this). Who would have expected forward and reverse DNS checks to take place…? 😎
I then wanted to make it easy for myself again so that I would not have to reconfigure the appliance, but even the subsequent manual entry into the DNS server did not lead to the desired result, as the appliance was also working with the host name “localhost”, which cannot be resolved so easily via DNS…
Here’s the error:

Failed Connectivity Tests because of missing DNS entries

So without further ado, I installed the appliance again, this time with a fixed IP address and previously created DNS entries. This can be done quickly and the installer has also remembered most of the settings! Top! 👍

Set Static IP for the Appliance

Quickly select the desired time synchronisation method (host or NTP server)…

…and finally an NTP Server

…and the deplyoment starts…

…and is ready after about 8 minutes in our Lab environment:

Deplyoment complete, Hit Launch to open the a Browser with next steps

After clicking on Launch, the browser of our choice opens and shows us the web interface of the Cloud Gateway, or can be found manually at https://<vmwarecloudgateway>:5480/gw-platform/

Click on Get Started:

The next step is to establish the connection between Cloud Gateway and VMware Cloud:

Step 1, Connect Cloud Gateway to VMware Cloud
Login Credentials needed for the Appliance

The cloud gateway will now run a few connectivity tests (for those who read closely, this didn’t work before with my first DHCP setup).
The latencies seem relatively high, but I may nevertheless press Next:

Connectivity Tests

The connection between Cloud Gateway and VMC must also be confirmed by a 6-digit code, after which VMware computes very hard…:

Approval with 6-Digit code
Calculating….

The excitement continues. This step took about 10 minutes:

…and was successful afterwards. Cloud Gateway and VMC are connected, now the vCenter(s) that are to be linked to the Cloud Gateway are missing. This will happen in the next steps:

Step 2: Connect vCenter Servers
Name/IP, User and Password for the vCenter
Next!
…and Acknowledge sending data to VMC

A few seconds later it is shown as Connected:

VMware Cloud vSphere+ and vSAN+ Dashboard

Back in my VMware Cloud Dashboard, I see my vSphere+ and vSAN+ tile:

Then let’s “launch” the whole thing and take a look at the new functions.

In the Inventory there is a nice overview of my “SDDCs”. There is also a Detail View, and the possibility to update the connected vCenter (assuming a valid subscription):

Inventory
Sorry, no subscription for you! 👎

In the Infrastructure Operations menu item, I am presented with an overview of the events and security analysis (such as enabled SSH):

Infrastructure Operations

The Desired State Profile makes it possible to define a certain state (by importing directly from the vCenter or via JSON file) and then assign it to one or more vCenters. This can then be checked regularly (automatically) and in this way ” discrepancies” can be detected and avoided:

Desired State Profile

I found this quite interesting, but there is certainly more to it: Apart from just viewing the VMs, it is also possible to create a VM from the VMC interface:

Create Virtual Machine Part 1
Create Virtual Machine Part 2

Conclusion

vSphere+ (and vSAN+) are certainly a very nice idea, but in my opinion the advantage (or the goal?) currently lies more in the change to the licensing model that one is getting involved with, because there is a change to subscriptions. Extra functions for vSAN (aka vSAN+) are not yet to be discovered, and the extras for vSphere+ are also manageable or still very much in line with other services such as Skyline Advisor. VMware has taken a first step here and shows with some first examples (vCenter Update/Upgrade and the creation of simple VMs via VMC) what a truly centralised administration of a strongly fragmented vCenter environment could look like.
In addition, through the connection to VMC, interaction with other VMC services is also imaginable in the future (such as VMware Cloud Disaster Recovery, Hyperscalers,…) or already in sight, as mentioned in this VMware blog article.

To complete the picture, you can also read the VMware documentary itself here:
https://docs.vmware.com/en/VMware-vSphere+/services/vsphereplus-using-managing/GUID-7D5545D9-6E35-48AA-88AE-BE549940344F.html

🐵

VMware NSX ALB / AVI Certificate Signing Request (CSR) with Microsoft CA

Reading Time 2 Minutes

Since I was dealing with VMware NSX Advanced Load Balancer (or NSX ALB aka AVI Loadbalancer, take your pick!) in connection with vSphere with Tanzu in our ITQ Lab environment, I also wanted to make it trustworthy with an “official” certificate and replace the “Self Signed Certificate”.

In the Lab environment, a Windows Certificate Authority (CA) and the CA Web Enrollment already exist on a Windows 2019 VM. I will not go into detail about the installation of the CA; here I followed an article on the VirtuallyThere Blog.

In addition, I have created a Certificate Template according to VMware specifications. In the future, I would like to use this for all VMware product deployments in the lab. I have written another blog article about this (Creating a Microsoft CA Template for vSphere 6.x/7.x), or if you want to read it directly from VMware, here is the KB article: Creating a Microsoft Certificate Authority Template for SSL certificate creation in vSphere 6.x/7.x

Continue reading

Creating a Microsoft CA Template for vSphere 6.x/7.x

Reading Time 2 Minutes

In order to be able to create uniform certificates that are signed by a Microsoft CA in our lab environment in a meaningful and “VMware compliant” way, it was necessary to create a Certificate Template in advance.

The following instructions are based on the VMware KB article Creating a Microsoft Certificate Authority Template for SSL certificate creation in vSphere 6.x/7.x

Continue reading