Nexus 1000v ESXi Uplink Port Greyed Out

27 Sep 2013 by Simon Greaves

If you find that when you look in vCenter at the uplinks for a Nexus 1000v distributed switch and you see that they are greyed out or the error message port blocked by admin this can be a sign that the host is not communicating correctly with the Nexus switch. Ports blocked by admin.  (Names blanked out to protect the innocent.)

There are two parts to this fix, the first if you can’t get online with virtual machines and the second if you can.  Scroll down to resolution part two to read more.

Resolution part one

Each ESXi host has a VEM, a Virtual Ethernet Module running on it and the VEM talks to the VSM, the Virtual Supervisor Module over the control and packet VLAN networks to discover what the configuration of the ports are and what portgroups and VLANs etc are assigned to the uplinks of the host.

It is worth checking that the correct VEM is installed as the Nexus can communicate only with a VEM that is on the same or earlier version than the VSM.  Don’t install a later version, even if it contains patches and hot-fixes as the Nexus wont see it and although it will appear that it is connected to the switch, albeit with greyed out uplinks, virtual machines won’t be able to communicate with any other VMs external to this host.

You may find that logging into the Nexus you see that the host is missing and that the error logs state the following error message:

%VEM_MGR-2-VEM_MGR_NOT_BC: Module cannot be inserted because it is not backward compatible

Now the error states backwards compatible but in fact the error is that it is not forwards compatible.  Great huh?!

If you have an environment with existing hosts connected to the Nexus switch you can run the following command on the Nexus switch to compare the version of the VEM currently installed on the existing hosts and then compare that to the version installed on the host with connectivity issues.

# show module

The output will be similar to the following:

Mod Ports     Module-Type                        Model     Status
--- ----- ------------------------------- ------------------ ------------
1    248     Virtual Ethernet Module              NA             ok

Mod    Sw                  Hw
---  ------------------  ------------------------------------------------
1    4.2(1)SV1(5.2)      VMware ESXi 5.0.0 Releasebuild-721882 (3.0)

The information under the Sw section shows the module installed. I suggest you then go to one of the working hosts and type the following.

# esxcli software vib list

This will list out all the installed modules.  Look for the one made by Cisco with the same revision number and note down the particular build number, then compare this to the one in the host that has the greyed out uplink and check they are the same.  If they are not you can download the same version either from the web interface of the Nexus switch (management IP address/HTTP service, if configured/enabled) or by downloading it from either the VMware downloads site or the Cisco website.

Please note both these sites require an active support subscription for you to download them.

Once downloaded remove the host from the Nexus dvSwitch in vCenter, SSH to the host and remove the existing Cisco module and install the new one using these commands.

# esxcli software vib remove -n name:version

# esxcli software vib add -d /PathToVIBModule.zip

Finally list out the installed VIB modules to make sure you now have the correct one installed.

# esxcli software vib list

Assuming all the above is done you should now find that the error messages on the Nexus have gone away and that the host is no longer missing on the switch leaving you free to connect the host back to the switch and to test VM network connectivity.

Resolution part two

Assuming that you are able to get online with VMs now attached to the switch portgroups you may still find that the icon is greyed out.  The fix for this is a simple one.

Open vCenter>Inventory>Networking select the uplink icon for the one that is greyed out and select the port tab.  Now select the host and click start monitoring port state.  If it is running already stop monitoring port state and start monitoring port state.  Viola.  Hopefully your problem has now gone away.

Tagged with: vSphere vCD

Comments are closed for this post.