I was recently updating my home lab and ran into an issue with my NICs as VMware no longer has the Realtek 8168 drivers embedded in the ESXi installer for 5.5/6.0.  I tried the old method of injecting the driver into the ISO and that proved unsuccessful since the driver is blacklisted — with the release of vSphere 6.0 VMware has implemented a driver blacklisting feature.  Fortunately you can bypass this feature to get your home lab up and running: Read the rest of this entry »

Having problems?  You should check the logs for anything obvious–

vCenter Server:

• Windows Server 2003 and earlier - %ALLUSERSPROFILE%\Application Data\VMware\VMware VirtualCenter\Logs\
• Windows 2008 and later - %ALLUSERSPROFILE%\VMware\VMware VirtualCenter\Logs\
• Linux vCenter Server 5.0 Virtual Appliance  – /var/log/vmware/vpx

ESXi Hosts (v3.5 – v5.x):

• /var/log/messages – VMkernel – Devices, drivers, VM stuff, as well as hostd and vpx logs
• /var/log/vmware/hostd.log – Host service stuff – Host & VMs Tasks & Events, vSphere Client stuff, vCenter Agent (vpxa) and SDK stuff
• /var/log/boot-logs/sysboot.log – More VMkernel startup including modules and other boot bits
• /var/log/vmware/vpx/vpxa.log – vCenter (vpxa) agent logs & hostd logs
• /var/log/vmware/fdm/* – HA Logs – vCenter 5.0 and later
• /var/log/vmware/aam/* – HA Logs – vCenter 4.1 and older

ESX Hosts: (v3.x and v4.x):

• /var/log/messages – Service Console Kernel – Services stuff will be here
• /var/log/vmkernel – VMkernel – Devices, drivers, VM stuff
• /var/log/vmkwarning – VMkernel Warning messages — these should also be in the vmkernel log file
• /var/log/vmksummary – Host startup/shutdown stuff, hourly log output showing uptime, # of VMs running, service console resource usage
• /var/log/vmware/hostd.log – Host service stuff – Host & VMs Tasks & Events, vSphere Client stuff, vCenter Agent (vpxa) and SDK stuff
• /var/log/boot-logs/sysboot.log – More VMkernel startup including modules and other boot bits
• /var/log/vmkiscsid.log – Logs for the Software iSCSI Client
• /var/log/vmware/vpx/vpxa.log – vCenter (vpxa) agent logs & hostd logs
• /var/log/vmware/fdm/* – HA Logs – vCenter 5.0 and later
• /var/log/vmware/aam/* – HA Logs – vCenter 4.1 and older

This is another updated version with more Intel removal goodness…see my previous post for more information on Automated P2V Cleanup: Remove/Uninstall Dell OpenManage & Broadcom/Intel Drivers Updated so it removes all the new Dell/Intel/Broadcom stuff released since the last update.

This Removes:

• Dell OpenManage Server Administrator
• Broadcom Drivers and Management Applications
• Broadcom NetXtreme II Driver Installer
• Intel(R) PROSet for Wired Connections
• Intel(R) PRO Network Connections

Read the rest of this entry »

Pre-P2V

• Cleanup Irrelevant Data
  • Don’t bring over that downloads folder on every server full of drivers, hotfixes, patches, service packs, etc.  Remember storage is now shared among many VMs now, try not to duplicate things like that.
• Stop transactional services during P2V
  • MSSQL, mySQL, etc.
• Disable Windows Updates
  • You don’t want your source to reboot due to automatic updates mid-P2V
• Do not resize volumes during P2V
  • Resizing volumes will increase P2V time significantly because instead of doing a block-level copy the convert would have to use file-level copy
• Set Target disks to Thin
  • Take advantage of thin provisioning–make sure you have vCenter Alarms setup to alert you when datastores near capacity

Post-P2V

• Move the Page File to a seperate volume (if it isn’t already)
  • If you are moving away from traditional backup methods to new vStorage methods you can’t exclude the page file from backups typically–however if it’s on a dedicated volume (separate vmdk) you can typically exclude it then
• Don’t Team NICs
  • You should build redundancy at the vSwitch level and use a single NIC for the VM (having a separate NIC connected to another vSwitch for backups, or Internal/External Interfaces is still ok)
• Cleanup
  • Get rid of all the extra software/agents physical servers require–Dell OpenManage, HP Insight, Broadcom/Intel Teaming Utilities – If your using Dell Servers use my Cleanup Script to automate the process

We needed a fast way to provision 180+ ESXi servers to boot from SD Cards, ESXi 4.1 introduced Scripted Installation to ESXi (the feature has been in ESX classic for awhile now)

However it is limited:

• Installation on USB Devices (SD cards, flash drives, etc.) isn’t currently supported.

Uhoh, our plan is to use SD Cards in Dell PowerEdge T610s and Dell PowerEdge 29xx (I’ll explain how on the 29xx in a later post) for our installs.  This isn’t good…

We figured we could build a golden image and clone our SD cards from that, but cloning the SD cards proved difficult.  I tried a lot of different apps and nothing seemed to work–we had trouble with DD even.  Then I stumbled across this blog post from Luke: Fast deployment of vSphere ESXi 4.0 running on a 1GB SD-Card

Luke recommended WinImage to do this cloning, I still couldn’t get WinImage to work at first but a quick email to Luke cleared up my troubles (thanks again Luke)

It was good to see my process to create the gold image was similar to how Luke did his just a few minor variations:

Creating the Gold Image

• Use a spare box for these steps–
• Install the Dell customized ESXi 4.1 onto the SD Card
  • The Dell customized ESXi isn’t actually required–from what I’ve read it simply updates support links and other textual things to refer to Dells support pages for various things…but whatever.
• Add the host in vMA so I can install OpenManage 6.3.0
  • OpenManage for ESXi is provided as a VIB and I choose to install via vMA
• Set root password
• Configure CIMOEMProvidersEnabled
• Configure NTP
• Add your license key–even if it’s ESXi free VMware will provide a license key — otherwise you’ll start the evaluation clock ticking…
• Create custom roles
  • By default there are only three roles: No access, Read-only, and Administrator
  • I create an additional role, Virtual machine user with basic access to work on VMs (Power on/off, Remote Control, Mount Discs, etc.)
• Ensure networking is set for DHCP
• Ensure no datastores exist
• Shutdown & yank the SD card out of the host
• Clone to a file using WinImage
• Clone from a file to a new SD card using WinImage
• Test
• Profit $$$

So you P2V’d a Linux box (CentOS, RedHat, Oracle, etc.) that is using LVM and now when it boots you get:

Reading all physical volumes. This may take awhile…
Volume group “VolGroup00” not found
Unable to access resume device (/dev/VolGroup00/LogVol01)
mount: could not find filesystem ‘/dev/root’
setuproot: moving /dev failed: No such file or directory
setuproot: error mounting /proc: No such file or directory
setuproot: error mounting /sys: No such file or directory
switchroot: mount failed: No such file or directory
Kernel panic – not syncing: Attempted to kill init!

Well that sucks–basically initrd didn’t have the right drivers to access the file system.

Reboot the VM with the Linux distro ISO attached

Then it boots from the CD follow the instructions to enter Linux Rescue (usually you type “linux rescue” at the boot prompt)

Now we can rebuild initrd with the missing drivers.

Enter:

Type:

Now hit tab and it will auto complete then add the text that was autocompleted when you hit tab except for the .img

So your command should look like:

Hit enter and it will do it’s thing with a lot of output coming out, when it’s done type exit annd exit again and the VM will reboot automatically.  Should boot up just fine this time.

The official VMware KB 1006996 on virtualizing existing domain controllers recommends 4 options (simplified below):

1. Demote the DC via dcpromo, do the conversion, then promote the DC back again using dcpromo–With all that trouble why not just build a new one?
2. Cold Clone–Cool, except if I’m working remote I need a working out of band management solution to boot the DC with the ISO…or setup a PXE server to provide a bootable cold clone image to do this, regardless too much extra leg work.
3. Use Directory Services Restore Mode (DSRM) and do a hot conversion, cool, but now I need to reboot these guys and hope someone remembers the DSRM password, or reset it before hand…
4. Get rid of the old DC via dcpromo and build a new VM and promote it–no conversion at all, wait what?

Having to P2V over a hundred DCs we didn’t particually like any of these options, the KB focuses on the possibility of a corrupt NTDS.DIT if you don’t follow any of those recommendations.  For our first few we did the DSRM route:

Read the rest of this entry »

This is an updated version with more Intel removal goodness…see my previous post for more information on Automated P2V Cleanup: Remove/Uninstall Dell OpenManage & Broadcom/Intel Drivers

This Removes:

• Dell OpenManage Server Administrator
• Broadcom Drivers and Management Applications
• Broadcom NetXtreme II Driver Installer
• Intel(R) PROSet for Wired Connections
• Intel(R) PRO Network Connections

Read the rest of this entry »

Close to a year ago we proposed the idea to virtualize all the servers at our remotes sites.  It took a significant amount of time to prove the value of virtualizing these remote sites, build the business case, present it, finally get approval, then create the in-depth project plan.  As of today we have 10% of our sites virtualized.

One of the major decisions we made was to use ESXi Free–we had a lot of push back at first, but continued to explain any risk of using the free product was mitigated with our plan.  For the technical folks it was simple, ask them:  What does licensing ESXi get you?  Most people who were asked this didn’t know the answer–the big hit is vCenter.

No vCenter?

Yes thats right–no vCenter.  The biggest frown here is no centralized console to manage everything from.  To us that isn’t a big deal–we’ve been manging distributed systems for years without a centralized management console.  All the hosts are setup with  Active Directory authentication so Role Baesd Access Control (RBAC) is cake, and the staff that need access have just the access they need.

We do not have shared storage in the remote sites so we can’t really take advantage of clusters–and along with that DRS, DPM, HA, FT, etc.  as well as vMotion…

vCenter does provide all the hardware and performance monitoring, but we have that covered to…

What about monitoring?!

At the VMware level

The hosts are monitored by Veeam Monitor Free is a fantastic product that will give us the nice dashboard and aggregate alarms and performance from all the remote hosts.  It’s awesome that Veeam offers this product for free.  The only limitation we’ve really noticed thus far is email alerts of these alarms.

At the Hardware level

Hardware is monitored using Whatsup Gold! with a monitor against the DRAC for hardware status–Whatsup will generate the email alerts on any hardware issues.

I posted howto setup this monitor on the WhatsUp Gold Community site: Dell Server Hardware Monitor VIA DRAC (SNMP Monitor)

What about updates?

While vCenter Update Manager is nice we can patch the hosts  using the vSphere Management Assistant (vMA) appliance–look out for posts in the future on how we accomplish this.

Here we are about a month and a half into the actual conversion process!

I’ve already posted some of our obstacles in previous posts and how we avoided/got around them, but stay tuned for more–like how we actually handled provisioning 180+ ESXi hosts.

So how many times have you done a P2V only to have to go back and manually cleanup all the extra crud a physical server requires–Dell OpenManage Server Administrator(Dell OSMA), Broadcom Drivers and Management Applications, Intel Drivers and Management Applications, etc.  Now expand that to having to do over 500 P2Vs in a few month span and you find the need to automate….

Presenting a huge honking script to automate those pesky uninstalls, right now it’s only Dell specific, but I plan to expand it to other vendors like IBM & HP.

This Removes:

• Dell OpenManage Server Administrator
• Broadcom Drivers and Management Applications
• Broadcom NetXtreme II Driver Installer
• Intel(R) PROSet for Wired Connections
• Intel(R) PRO Network Connections

Read the rest of this entry »