In Part 1 of this series I covered the necessary concepts to understand Cisco’s VM-FEX. In this part I’ll cover how to go about configuration UCS Manager and performing the necessary Vmware integration.

Configuration of UCS Policies

First we need to configure the policies in UCS that will allow our server profile to support VM-FEX.

Dynamic vNIC Connection Policy

The first policy is related to  the vNICs that will be used for each VM. Create a Dynamic vNIC policy by looking under the Policies section of the LAN tab, Right-click and select to create a new policy. You’ll get the following screen in which you will need to fill out or select the relevant options.

The important options are:

Number of Dynamic vNICs – This is the number of vNICs that will be available for dynamic assignment to VMs. Remember that the VIC has a limit to the number of vNICs that it can support and this is based on the number of uplinks between the IOM and the FI. At least this is the case with the 2104 IOM and the M81KR VIC, which supports ((# IOM Links * 15) – 2)). Also remember that your ESXi server will already have a number of vNICs used for other traffic such as Mgmt, vMotion, storage, etc, and that these count against the limit.

Adapter Policy – This determines the vNIC adapter config (HW queue config, TCP offload, etc) and you must select VMWarePassThru to support VM-FEX in High Performance mode.

Protection – This determines the initial placement of the vNICs, either all of them are placed on fabric A or Fabric B or they are alternated between the two fabrics if you just select the “Protected” option. Failover is always enabled on these vNICs and there is no way to disable the protection.

BIOS Policies

You need ensure that all the BIOS virtualization options are enable on the blades so they can support VM-FEX. To do this create a BIOS policy and set the Virtual Technology (VT) option under the Processor section and all the options under the Intel Directed IO section to enabled.

While it seems like these would all be enabled by default, when I looked at a blade that uses the default BIOS policy with each option set to “Platform Default”, It had the Coherency Support set to disabled. So make sure to create the policy to ensure everything is set correctly.

Create UCS Service Profile

To create a service profile that supports VM-FEX you will need to create a service profile using the “expert mode”. You will need to select the Dynamic vNIC Connection Policy on the networking screen (Step 3) and select the BIOS policy on the Operational Policies screen (Step 8).

vNICs for the VM-FEX DVS

As mentioned in the concepts portion of this series, VM-FEX looks like a DVS to vCenter. All virtual switches  need to be configured with up-link interfaces so that traffic on the virtual switch can be sent to the physical world, but what about the VM-FEX?

With VM-FEX each VM is assigned to it’s own vNIC, so it seems logical that the DVS would not need any up-link interfaces, but this is not the case. If you do not configure any up-link interfaces on the VM-FEX DVS then your VMs NICs will be disconnected when they boot up and they will not be assigned a Veth on the UCS Fabric Interconnect.

So you need to make sure to configure a pair of static vNICs to be used as DVS up-links. The static vNICs used for up-links do not have to be configured for any specific VLANs or adapter policy. As far as I can tell they are not actually used for any traffic.

Assign Service Profile to a Blade

You should be able to see the dynamic vNICs under the network tab of the service profile once it’s assigned to a blade. You may need to boot the blade before they show up and it should look something like this.

Install ESXi 5.0 and Cisco VEM Software Bundle

Next you need install ESXi 5.0 on the blade. There is no Cisco or UCS specific ESX image or installer, just use the one you can download from VMware.

Once ESXi is installed you will need to install the Cisco VEM Software Bundle. You access the bundle from the UCS Manager launch web page.

This link will bring you to a page with the VIB that you will need to download to your ESXi server. Look for the ESXi 5.0 or later in the description and find the URL for cross_cisco-vem-v132-4.2.1.1.4.1.0-3.0.4.vib. You will need the URL of the VIB for the next step of installing the VIB on your ESXi server.

Once you have the URL for the VIB you will need to enable SSH on your ESXi server and SSH into the server. Then execute the command in the following example to install the VIB. Make sure to replace the URL in the example below with the URL of the VIB file on your UCSM.

~ # esxcli software vib install --viburl http://hq-demo-ucsm/cisco/vibs/VEM/4.1.0/VEM-4.1.0-patch01/cross_cisco-vem-v132-4.2.1.1.4.1.0-3.0.4.vib
Installation Result
   Message: Operation finished successfully.
   Reboot Required: false
   VIBs Installed: Cisco_bootbank_cisco-vem-v132-esx_4.2.1.1.4.1.0-3.0.4
   VIBs Removed:
   VIBs Skipped:

Configuring VMware Integration and Create VM-FEX DVS

You will need to enable the integration between vCenter and UCSM since the VM-FEX leverages VMware’s DVS.

Modify Extension Key

This is optional, but it makes the integration with vCenter easier to spot (You’ll see later) and is really simply to do, so I’m would recommend it.

In UCSM go to the VM tab and click on VMware. On the right side there will be a Modify Extension Key under actions. Click on this and enter a name for the UCSM vCenter integration. I just used the name of my UCSM.

VMware Integration Wizard

Next we will run the VMware Integration Wizard, which is launched by selecting Configure VMware Integration, below the Modify Extension Key link. This wizard will take you through the steps to integrate with VMware and create a VM-FEX DVS.

Step 1: Install Plug-in on vCenter Server

The first step involves installing the UCSM vCenter Plug-in on your vCenter server. When the VMware Integration Wizard is luanched the first screen will give you the option to export the plugin. Click the Export Button and save the Plug-in to your desktop.

Once the plug-in is downloaded go into your vCenter Client and import the plug-in by selecting Manage Plug-ins… found under the Plug-in’s menu.  At this point the Plug-in manager screen will launch and you will right click on a part of the manager screen without a plug-in and select New Plug-in. Browse to where you downloaded the Plug-in and select it. At this point a new plug-in should appear in the Plug-in Manager with the name that you entered when you modified the extension key.

In my example you can see that there are two plug-ins, one called hq-demo-ucsm and the other called Cisco-UCSM-<snip>. The Cisco-UCSM-<snip> plugin was form a previous UCSM integration where I did not modify the extension key.

Step 2: Define VMware DVS

The next step defines the information that will be used to create the DVS . This step’s screen looks like this.

You need to make sure that the vCenter info and Datacenter name matches what you currently have in your VMware environment.

The DVS folder name and DVS name can be what ever you would like, but obviously make it something that make sense to you. Also make sure that the DVS is set to Enabled.

Step 4: Define Port Profile

This next step is used to define the port profile that is to be used to create the port groups on the DVS.

Here you enter the name of the port profile, desired policies and VLANs. In general you are going to select one VLAN and make it native (untagged). If you select more then one VLAN then each vNIC will be configured as a trunk and the VM will have to be configured to do 802.1q.

The second part is the Profile Client information where you enter a name and select which DVS you want to create the port group on. In my config I just named the profile client the same as the port profile and selected the DVS I just created. You could choose “All” for the DVS and it would add the port group to any future VM-FEX DVS that is created.

 Step 5: Apply Configuration

In this step you just click Finish and all the configuration is applied. If you are able to switch over to the vCenter client you can see the DVS being created and configured.

Once all the configuration has completed you should see the new DVS under the Networking Inventory section of the vCenter client.

Step 6: Enable High Performance Mode

According to the documentation you are suppose to select high performance mode when using the VMware Integration Wizard, but I didn’t see the place to set it. I’ve run through the wizard a few times and I don’t think it is there. So instead I just changed the setting on the Port Profiles General tab.

VM-FEX Configuration Complete

If you followed along you should now have VM-FEX deployed to one or more of your ESXi 5.0 servers. In Part 3 of this series I’ll walk through the configuration of the DVS and VMs to support High Performance Mode.