Having problems?  You should check the logs for anything obvious–

vCenter Server:

• Windows Server 2003 and earlier - %ALLUSERSPROFILE%\Application Data\VMware\VMware VirtualCenter\Logs\
• Windows 2008 and later - %ALLUSERSPROFILE%\VMware\VMware VirtualCenter\Logs\
• Linux vCenter Server 5.0 Virtual Appliance  – /var/log/vmware/vpx

ESXi Hosts (v3.5 – v5.x):

• /var/log/messages – VMkernel – Devices, drivers, VM stuff, as well as hostd and vpx logs
• /var/log/vmware/hostd.log – Host service stuff – Host & VMs Tasks & Events, vSphere Client stuff, vCenter Agent (vpxa) and SDK stuff
• /var/log/boot-logs/sysboot.log – More VMkernel startup including modules and other boot bits
• /var/log/vmware/vpx/vpxa.log – vCenter (vpxa) agent logs & hostd logs
• /var/log/vmware/fdm/* – HA Logs – vCenter 5.0 and later
• /var/log/vmware/aam/* – HA Logs – vCenter 4.1 and older

ESX Hosts: (v3.x and v4.x):

• /var/log/messages – Service Console Kernel – Services stuff will be here
• /var/log/vmkernel – VMkernel – Devices, drivers, VM stuff
• /var/log/vmkwarning – VMkernel Warning messages — these should also be in the vmkernel log file
• /var/log/vmksummary – Host startup/shutdown stuff, hourly log output showing uptime, # of VMs running, service console resource usage
• /var/log/vmware/hostd.log – Host service stuff – Host & VMs Tasks & Events, vSphere Client stuff, vCenter Agent (vpxa) and SDK stuff
• /var/log/boot-logs/sysboot.log – More VMkernel startup including modules and other boot bits
• /var/log/vmkiscsid.log – Logs for the Software iSCSI Client
• /var/log/vmware/vpx/vpxa.log – vCenter (vpxa) agent logs & hostd logs
• /var/log/vmware/fdm/* – HA Logs – vCenter 5.0 and later
• /var/log/vmware/aam/* – HA Logs – vCenter 4.1 and older

This is another updated version with more Intel removal goodness…see my previous post for more information on Automated P2V Cleanup: Remove/Uninstall Dell OpenManage & Broadcom/Intel Drivers Updated so it removes all the new Dell/Intel/Broadcom stuff released since the last update.

This Removes:

• Dell OpenManage Server Administrator
• Broadcom Drivers and Management Applications
• Broadcom NetXtreme II Driver Installer
• Intel(R) PROSet for Wired Connections
• Intel(R) PRO Network Connections

Read the rest of this entry »

Pre-P2V

• Cleanup Irrelevant Data
  • Don’t bring over that downloads folder on every server full of drivers, hotfixes, patches, service packs, etc.  Remember storage is now shared among many VMs now, try not to duplicate things like that.
• Stop transactional services during P2V
  • MSSQL, mySQL, etc.
• Disable Windows Updates
  • You don’t want your source to reboot due to automatic updates mid-P2V
• Do not resize volumes during P2V
  • Resizing volumes will increase P2V time significantly because instead of doing a block-level copy the convert would have to use file-level copy
• Set Target disks to Thin
  • Take advantage of thin provisioning–make sure you have vCenter Alarms setup to alert you when datastores near capacity

Post-P2V

• Move the Page File to a seperate volume (if it isn’t already)
  • If you are moving away from traditional backup methods to new vStorage methods you can’t exclude the page file from backups typically–however if it’s on a dedicated volume (separate vmdk) you can typically exclude it then
• Don’t Team NICs
  • You should build redundancy at the vSwitch level and use a single NIC for the VM (having a separate NIC connected to another vSwitch for backups, or Internal/External Interfaces is still ok)
• Cleanup
  • Get rid of all the extra software/agents physical servers require–Dell OpenManage, HP Insight, Broadcom/Intel Teaming Utilities – If your using Dell Servers use my Cleanup Script to automate the process

So you P2V’d a Linux box (CentOS, RedHat, Oracle, etc.) that is using LVM and now when it boots you get:

Reading all physical volumes. This may take awhile…
Volume group “VolGroup00” not found
Unable to access resume device (/dev/VolGroup00/LogVol01)
mount: could not find filesystem ‘/dev/root’
setuproot: moving /dev failed: No such file or directory
setuproot: error mounting /proc: No such file or directory
setuproot: error mounting /sys: No such file or directory
switchroot: mount failed: No such file or directory
Kernel panic – not syncing: Attempted to kill init!

Well that sucks–basically initrd didn’t have the right drivers to access the file system.

Reboot the VM with the Linux distro ISO attached

Then it boots from the CD follow the instructions to enter Linux Rescue (usually you type “linux rescue” at the boot prompt)

Now we can rebuild initrd with the missing drivers.

Enter:

Type:

Now hit tab and it will auto complete then add the text that was autocompleted when you hit tab except for the .img

So your command should look like:

Hit enter and it will do it’s thing with a lot of output coming out, when it’s done type exit annd exit again and the VM will reboot automatically.  Should boot up just fine this time.

The official VMware KB 1006996 on virtualizing existing domain controllers recommends 4 options (simplified below):

1. Demote the DC via dcpromo, do the conversion, then promote the DC back again using dcpromo–With all that trouble why not just build a new one?
2. Cold Clone–Cool, except if I’m working remote I need a working out of band management solution to boot the DC with the ISO…or setup a PXE server to provide a bootable cold clone image to do this, regardless too much extra leg work.
3. Use Directory Services Restore Mode (DSRM) and do a hot conversion, cool, but now I need to reboot these guys and hope someone remembers the DSRM password, or reset it before hand…
4. Get rid of the old DC via dcpromo and build a new VM and promote it–no conversion at all, wait what?

Having to P2V over a hundred DCs we didn’t particually like any of these options, the KB focuses on the possibility of a corrupt NTDS.DIT if you don’t follow any of those recommendations.  For our first few we did the DSRM route:

Read the rest of this entry »

This is an updated version with more Intel removal goodness…see my previous post for more information on Automated P2V Cleanup: Remove/Uninstall Dell OpenManage & Broadcom/Intel Drivers

This Removes:

• Dell OpenManage Server Administrator
• Broadcom Drivers and Management Applications
• Broadcom NetXtreme II Driver Installer
• Intel(R) PROSet for Wired Connections
• Intel(R) PRO Network Connections

Read the rest of this entry »

Close to a year ago we proposed the idea to virtualize all the servers at our remotes sites.  It took a significant amount of time to prove the value of virtualizing these remote sites, build the business case, present it, finally get approval, then create the in-depth project plan.  As of today we have 10% of our sites virtualized.

One of the major decisions we made was to use ESXi Free–we had a lot of push back at first, but continued to explain any risk of using the free product was mitigated with our plan.  For the technical folks it was simple, ask them:  What does licensing ESXi get you?  Most people who were asked this didn’t know the answer–the big hit is vCenter.

No vCenter?

Yes thats right–no vCenter.  The biggest frown here is no centralized console to manage everything from.  To us that isn’t a big deal–we’ve been manging distributed systems for years without a centralized management console.  All the hosts are setup with  Active Directory authentication so Role Baesd Access Control (RBAC) is cake, and the staff that need access have just the access they need.

We do not have shared storage in the remote sites so we can’t really take advantage of clusters–and along with that DRS, DPM, HA, FT, etc.  as well as vMotion…

vCenter does provide all the hardware and performance monitoring, but we have that covered to…

What about monitoring?!

At the VMware level

The hosts are monitored by Veeam Monitor Free is a fantastic product that will give us the nice dashboard and aggregate alarms and performance from all the remote hosts.  It’s awesome that Veeam offers this product for free.  The only limitation we’ve really noticed thus far is email alerts of these alarms.

At the Hardware level

Hardware is monitored using Whatsup Gold! with a monitor against the DRAC for hardware status–Whatsup will generate the email alerts on any hardware issues.

I posted howto setup this monitor on the WhatsUp Gold Community site: Dell Server Hardware Monitor VIA DRAC (SNMP Monitor)

What about updates?

While vCenter Update Manager is nice we can patch the hosts  using the vSphere Management Assistant (vMA) appliance–look out for posts in the future on how we accomplish this.

Here we are about a month and a half into the actual conversion process!

I’ve already posted some of our obstacles in previous posts and how we avoided/got around them, but stay tuned for more–like how we actually handled provisioning 180+ ESXi hosts.

So how many times have you done a P2V only to have to go back and manually cleanup all the extra crud a physical server requires–Dell OpenManage Server Administrator(Dell OSMA), Broadcom Drivers and Management Applications, Intel Drivers and Management Applications, etc.  Now expand that to having to do over 500 P2Vs in a few month span and you find the need to automate….

Presenting a huge honking script to automate those pesky uninstalls, right now it’s only Dell specific, but I plan to expand it to other vendors like IBM & HP.

This Removes:

• Dell OpenManage Server Administrator
• Broadcom Drivers and Management Applications
• Broadcom NetXtreme II Driver Installer
• Intel(R) PROSet for Wired Connections
• Intel(R) PRO Network Connections

Read the rest of this entry »

Took me awhile to get what should have been a simple Linux P2V done at work!

This stuff wasn’t clearly stated anywhere–I couldn’t find anyone with similar issues, so I’m posting this up hoping to help someone else out in the future.  Working with VMware they pointed out the lack of support for dvSwitches after passing through a few levels of support their.

Error: Unable to connect to the Converter helper server on the target virtual machine

Cause: VMware does not support the usage of dvSwitches for Linux Conversions!  Make sure the network you select isn’t a dvSwitch

Error: Unable to obtain the IP address of the target virtual machine running the Converter helper server.

Cause: Related to the above error, relying on DHCP while using a network that is a dvSwitch for a conversion will get you the above error.

FAILED: MethodFault.summary

The last task before removal will be something like: Connecting the Converter helper ISO image to the target virtual machine.

Cause: ESX(i) 4.1 is not supported at this time by VMware Converter–this was with version 4.0.1–the latest version at the time this post was made.

Leave a comment if these notes helped you out or if you have any other uncommon errors you ran into.

Something cool for your ESX/ESXi labs–Rather then relying on using the VIC (VMware Infrastructure Client) to remote control your VMs you can enable VNC access to them.

To do this power off the VM and add the lines below to the VMs .vmx file, you will need to choose a different port for each VM you wish to do this on.

remotedisplay.vnc.port=”5900″

remotedisplay.vnc.enabled=”true”

If your using ESX you can password protect the VNC connection by adding this line:

remotedisplay.vnc.password = “password”