Archive for July, 2013

VCAP-CIA objective 2.2 – Manage vSphere storage resources

July 28th, 2013 No comments

The blueprint states the following skills needed to cover this objective.

  • Decommission storage
  • Create and manage storage profiles

The goal to decommission storage can prove to be a tricky one when done wrong, as this might get your host into an APD state. APD handling has been improved alot with ESXi 5.1 which is used in the exam but should you should still try to remove the storage correctly in order to not risk any point loss or even worse a host loss during your exam.

The following kb article shows the correct procedure to use which includes unmounting the datastore and detaching the device. If the storage device actually is an NFS share all that is needed is the unmount.

Unmounting a LUN or detaching a datastore/storage device from multiple ESXi 5.x hosts



This does cover the vSphere side of things. If asked to decommission a datastore from vCloud Director you will need to disable the datastore and remove it from all Provider vDCs. This can be achieved by editing the according storage profile for the datastore to not containing it anymore. If the “* (Any)” profile is used it will be sufficient to remove the datastore on the vSphere level. In the screenshots below we are going to disable the iSCSI16GB datastore, so no new VMs can be deployed to the datastore anymore. Afterwards we are going to remove it from the Provider vDC by editing the storage capabilities of the datastore on the vSphere level so it is not contained in the according iSCSI storage profile anymore.






The creation and management of storage profiles is done at the vSphere level. Once you have created and enabled storage profiles they can be assigned to a Provider vDC and Organization vDCs. This process is described in the vCloud Director Administrator’s Guide on pages 49 and 63 in the English version. Additional information can be found in the following blog posts.

The first step is to enable storage profiles on the clusters or hosts used for vCloud Director.



If your storage is not VASA (vSphere Storage APIs – Storage Awareness) capable you will need to create user defined capabilities first.



The third step is assigning these capabilities to your datastores.


The final step is actually creating the storage profiles.



Your datastores are now mapped to the profiles by the assigned capabilties which covers the vSphere side of things. You are now ready to use these storage profiles in vCloud Director. You will be able to choose them during the creation of a Provider vDC. If you want to edit an already existing Provider vDC to use some newly created storage profiles you can do so in the Manage and Monitor tab.


Categories: Certification, VCAP Tags:

VCAP-CIA objective 2.1 – Add vSphere compute resources to vCloud Director

July 27th, 2013 No comments

The blueprint states the following skills needed to cover this objective.

  • Add new vCenter servers to vCloud Director
  • Prepare/unprepared hosts in vCloud Director
  • Add ESXi hosts to vCenter
  • Manage ESXi hosts and DRS resource pools in vCenter

All 3 objectives of this section of the blueprint will require knowledge of the underlying vSphere architecture and less vCloud Director itself.

Before you can add a new vCenter Server to vCloud Director you will need to deploy a vShield Manager Appliance for that vCenter and establish the connection. Once that is done the procedure on how to add the vCenter to vCloud Director is described in the vCloud Director Administrator’s Guide on pages 20 and 21 in the English version.

There are several ways to add a vCenter Server but the one requiring the least clicks should be to just click “Attach a vCenter” and enter the correct data.


You will need to provide the DNS name or IP of the vCenter Server, the port the vpxd service is listening on, 443 by default, a vCenter administrator and its password, a display name for vCloud Director, an optional description and the URL to the Web Client to open vSphere objectives when right clicking them in vCloud Director.


For the vShield Manager appliance you will also need the IP or DNS name and an admin user and the password, admin and default when nothing was changed,


When everything was successful you will find the added vCenter in the Manage & Monitor tab in the vSphere Resources.


Preparing a host is described on page 102 in the English version of the vCloud Director Administrator’s Guide. To be able to prepare a host you must make sure the host is not in maintenance mode. You also cannot prepare hosts if you do not have a provider vDC. If the host is part of a cluster of a provider vDC you will find it at the Manage & Monitor tab in the left pane under the Hosts option. Simply right clicking the host, choosing “Prepare Host” and providing root credentials completes the process.

The host is taken into maintenance mode in the vSphere layer, an agent is installed on the host, the maintenance mode is ended and the host will be able to serve as a compute resource for the cloud environment from now on. To reverse the process simply right click the host again, select “Unprepare Host” and wait until the process is finished.



Adding a host to vCenter Server is a pretty straight forward task which can be done via the Classic Client or the new Web Client. All you need to know are the root credentials for that host and the target cluster it should be put in. As the exam allows only for limited time and performance could be slow due to the exam being hosted offsite I would advise using the Classic Client for every action possible to do with it as it performs a little faster than the Web Client. I will still show a screenshot on how to add a host in the Web Client in case there is no access to the Classic Client in the exam.


The goal to manage hosts and resource pools is extremely vague and could mean alot of things. All what is needed can be found at the “Manage” tab or “Configuration” tab depending on what client you are using. You can manage a resource pool by right clicking on it and choosing “Edit settings”. You should make yourself familiar with the concepts of shares, limits and reservations again.

To get an overview of what could be asked read the vCloud Director Administrator’s Guide pages 99 – 104 in the English version which includes the following tasks.

  • Register vCloud Director with a vCenter Server
  • Modify vCenter Server Settings
  • Reconnect a vCenter Server
  • Enable or Disable a vCenter Server
  • Remove a vCenter Server
  • Prepare and Upgrade a vCenter Server Attached to vCloud Director
  • Modify vShield Manager Settings
  • Enable or Disable an ESX/ESXi Host
  • Move Virtual Machines from one ESX/ESXi Host to Another
  • Prepare or Unprepare an ESX/ESXi Host
  • Upgrade an ESX/ESXi Host Agent
  • Repair an ESX/ESXi Host
  • Enable or Disable a Datastore
  • Remove a Datastore
Categories: Certification, VCAP Tags:

VCAP-CIA Objective 1.4 – Configure Alarms and Notifications

July 27th, 2013 No comments

The blueprint states the following skills needed to cover this objective.

  • Configure SMTP and notification settings
  • Configure warning alerts
  • Create System maintenance message

SMTP settings can be configured on vCenter Chargeback Manager and vCloud Director using the GUI. The screenshots below show where the settings can be entered.




The way to configure the system warning alerts is described in the vCloud Director Administrator’s Guide on pages 51 and 103. Both pages describe the same procedure with a different way to get to the datastores. There are 2 thresholds that can be set, a yellow and red one. When these thresholds are crossed vCloud Director will send out an email warning about the low disk space automatically.


The configuration of a system maintenance message is described in the following kb and will be shown in the screenshots below. You can also find this information in the vCloud Director Administrator’s Guide on page 96 and 97 in the English version.

Configuring the VMware vCloud Director cell maintenance message


Categories: Certification, VCAP Tags:

VCAP-CIA Objective 1.3 – Maintain vCloud using command line tools

July 22nd, 2013 No comments

The blueprint states the following skills needed to cover the objective.

  • Manage and maintain vCloud Director cells using the cell management tool
  • Install and manage a vCloud Director installation using the configure script
  • Manage vCloud services using Red Hat command line tools including service, chkconfig and netstat
  • Collect logs for troubleshooting using the support script

The usage of the cell management tool is explained in the vCloud Director Installation and Upgrade Guide on pages 35 – 42 in the English version. It is located in the /opt/vmware/vcloud-director/bin directory on each cell and requires root credentials on the target server system administrator credentials for vCloud Director.

The cell tool offers different sub commands, the first bulletin of the blueprint configurable via the “cell” sub command. We will also have a short look at the other commands.


There seems to be only 1 particular kb article on how to use the cell management tool for quiescing a cell before shutting it down.

Using the vCloud Director Cell Management Tool to quiesce and shut down a server

An example output can be seen in the following screenshot.


The next sub command is “dbextract” which can be used to dump database contents without the need of a database management tool. There are several switches like “-exportSettingsFile” to further specify what exactly the dump will contain. Examples are in the /opt/vmware/vcloud-director/etc directory. If you want a full database dump you can use the syntax in the screenshot, just be sure to have a valid directory as the destination as the tool will not create one for you and will fail if the path does not exist.


The next sub command is “certificates” which allows you to replace SSL certificates on the cell. Remember that a restart is needed after replacing the certificates. As there is a whole objective dedicated to vCloud Director certificates I will only show the help output of this and the next sub command.


You can also create new SSL certificates using the “generate-certs” sub command. This might be an easier way than actually using the keytool.


Last but not least you also have the possibility to recover a lost admin password as long as you can remember the database password using the “recover-password” sub command.


This should cover the cell management tool, so we can move on to the second bulletin, the configure script which is also located in the /opt/vmware/vcloud-director/bin directory.

This binary can be used to update the vCloud Director cell configuration either by typing in the information manually or using the switch -r in combination with a response file as was covered in the previous objective. You can find an example of re configuring the cell using the manual way.


This leads us to the third bulletin. Which is not VMware specific at all but involves basic Linux administration skills. The service tool is used to run scripts located in /etc/init.d in a controlled way, to start, stop and restart services loaded by these scripts or give you an indication if the service is actually running. You can find a man page for the service command at or by simply typing “man service” via connected to the vCD cell.


Auto start options for services are controlled by the “chkconfig” command. The most important switches would be –list, –add, –del, on and off. More information can be found in the according man page, e.g.

To find out if the expected service is listening on a specific port you can use the netstat command in conjunction with the ps command. Additional parameters can be found in the man page as well.


We are listing on and with a VMware vCD process which is a good sign. If you see anything else in here and your cell fails to start you might want to check with chkconfig if the according process is automatically starting and take it off the auto run list to prevent it from breaking your cell.

Even though it is not mentioned in the blueprint you might also want to read up a little bit on iptables, the default firewall used in RHEL. The man page can be found at

The last bulletin is rather easy again. The process is described in the following kb article and shown in the last screenshot which also concludes this objective for today.

Collecting diagnostic information for VMware vCloud Director 1.x / 5.1.x


Categories: Certification, VCAP Tags:

VCPA-CIA Objective 1.2 – Configure vCloud Director for scalability

July 21st, 2013 5 comments

The blueprint states the following skills needed to cover this objective:

  • Generate vCloud Director response files
  • Add vCloud cells to an existing installation using response files
  • Set up vCloud Director transfer storage space
  • Configure vCloud Director load balancing

A response file is automatically created after the installation of the first cell in /opt/vmware/vcloud-director/etc and is called The process to add a vCloud cell to an existing installation is described on page 29 – 30 in the English version of the vCloud Director Installation and Upgrade Guide.

The procedure is also described in the following kb article.

Installing VMware vCloud Director software on additional servers

We will need access to the response file during the installation so this file could be copied to the transfer storage which needs to be mounted to the additional cells anyway and can be deleted after the installation process.

Before we continue with the server group installation we should do a couple of things.

  1. Create a DNS entry for the new cell as the services will not start otherwise
  2. The transfer storage NFS share should already be set up, if not here is what I did in my lab environment, the async parameter in the export config costs reliability but enhances the speed.
    • add disk – create a partition with cfdisk – create a filesystem with mkfs.ext3 or mkfs.ext4
    • create a mountpoint, e.g. /nfs/datastore1
    • edit /etc/fstab to auto mount the filesystem, e.g. /dev/sdb1 /nfs/datastore1 ext3 defaults 0 0
    • edit /etc/exportfs to export the share, e.g. /nfs/datastore1 *(rw,async,no_root_squash)
    • service nfs restart
  3. Edit the /etc/fstab file to auto mount the transfer storage on every cell
  4. Check permissions on the transfer storage, these should be user and group vcloud, if they are not set them with “chown -R “vcloud:vcloud” /opt/vmware/vcloud-director/data/transfer”
  5. Check permissions on the installation binary are set to executable, otherwise change them with “chmod u+x installation-file” and install the vCloud director binaries
  6. Generate the proper SSL certificates using the keytool
    • /opt/vmware/vcloud-director/jre/bin/keytool -genkey -keystore /opt/ssl/certificates.ks -storetype JCEKS -storepass passwd -keyalg RSA -validity 731 -alias http
    • /opt/vmware/vcloud-director/jre/bin/keytool -genkey -keystore /opt/ssl/certificates.ks -storetype JCEKS -storepass passwd -keyalg RSA -validity 731 -alias consoleproxy
    • /opt/ssl was the path I chose to save the keystores, you might do or be requested otherwise in the exam
  7. Run the configure script with the -r option pointing at the, e.g. /opt/vmware/vcloud-director/bin/configre -r /opt/vmware/vcloud-director/data/transfer/
  8. If you are done with all the cells delete the from the transfer storage directory






This leaves the load balancing configuration on the vCloud Director side as I would suppose the actual load balancer configuration should not be part of the exam itself as it is not covered in the Installation and Upgrade guide which is referenced in the blueprint. If you need more information on the actual load balancer configuration I will post some links to different blogs showing how to configure pfsense, F5 and vShield Edge to be an actual load balancer for vCloud Director.

What is relevant for the exam according to the blueprint can be found at Administration -> System Settings -> Public Addresses as can be seen in the screenshot below.


This concludes the second objective of the VCAP-CIA, as promised here are the links for some example load balancer configuration guides and best practices.

Categories: Certification, VCAP Tags:

VCAP-CIA Objective 1.1 – Configure centralized logging

July 19th, 2013 No comments

The blueprint states the following skills needed to cover this objective:

  • Determine use cases for and configure local and syslog options for vCloud Director
  • Configure and administer logging options for VMware vShield™ Manager
  • Configure Log4j options for vCloud Director Tools

The first and third goal are described in several kb articles but does not really have a good description in the official documentation. There is a nice explanation in the VMware vCloud Architecture ToolKit 2.0 but this example does not seem to have made it to version 3.1.

Pages 25 – 27 of the English version of the vCloud Director Installation and Upgrade Guide seem to be the only spot in the official product documentation mentioning the setup of a syslog server during the initial setup of a cell. This will be saved into the response file for the other cells to use.

If this step was omitted during initial setup it can be set in the file located in /opt/vmware/vcloud-director/etc.

Just add the proper syslog server after “ =” and restart the vcd service.

The VMware knowledge base describes how to activate the shipment of additional log files in the configuration file.

Enabling Centralized Logging in VMware vCloud Director 1.5.x / 5.1.x

Below you can find pictures of the configuration file in my environment, a TCP dump from the cell proving that it is now sending logs to the syslog server and the actual logs on the syslog server itself.






Further information about the threshold levels and local logging options can be found in the following kb article.

Configuring logging for VMware vCloud Director cells

Configuring a syslog server for vShield Manager is described in the English version of the vShield Administration Guide on page 19. There doesn’t seem to be a kb article explaining the process as this is really rather simple. Open up the vShield Manager configuration and pop in the correct values at the configuration pane under Settings & Reports.


Configuring the log shipping for firewall rules is done in the vCloud Director web interface. Just go to administration -> system settings -> general.


This should conclude the first objective of the VCAP-CIA, if you are interested you can find an even more advanced setup example in the vCAT 2.0 Tool.

The interesting part for this objective starts on page 41 in the English version.

Categories: Certification, VCAP Tags:

Why DNS is important…

July 19th, 2013 2 comments

I finally managed to actually configure all the shared storage options I needed (and guess the opendedup appliance will need to make way for a less RAM and CPU consuming FreeNAS or homebrew NFS appliance).

VSM setup was extremely straight forward and also the configuration of the first vCloud Director cell looked very promising after I enabled connections from the public network in the Windows firewall on Port 1433…

Still I would only see a blank screen when trying to connect to the vCloud Director landing page.


I never had a look at vCloud Director logs so far but there aren’t that many in the logs directory, so I took an educated guess.

2013-07-16 22:04:25,679 | INFO | pool-inittracker-1-thread-1 | ServiceInitializationTracker | Application initialization detailed status report: 0% complete
com.vmware.vcloud.common-util Subsystem Status: [WAITING]
com.vmware.vcloud.api-framework Subsystem Status: [WAITING]
com.vmware.vcloud.consoleproxy Subsystem Status: [WAITING]
com.vmware.vcloud.common-vmomi Subsystem Status: [WAITING]
com.vmware.vcloud.jax-rs-activator Subsystem Status: [WAITING]
com.vmware.pbm.placementengine Subsystem Status: [WAITING]
com.vmware.vcloud.vim-proxy Subsystem Status: [WAITING] Subsystem Status: [WAITING] Subsystem Status: [WAITING]
com.vmware.vcloud.fabric.compute Subsystem Status: [WAITING] Subsystem Status: [WAITING]
com.vmware.vcloud.backend-core Subsystem Status: [WAITING]
com.vmware.vcloud.ui.configuration Subsystem Status: [WAITING]
com.vmware.vcloud.imagetransfer-server Subsystem Status: [WAITING] Subsystem Status: [WAITING]
com.vmware.vcloud.jax-rs-servlet Subsystem Status: [WAITING]
com.vmware.vcloud.ui-vcloud-webapp Subsystem Status: [WAITING]
2013-07-16 22:04:32,928 | ERROR | SpringOsgiExtenderThread-3 | StdSchedulerFactory | Couldn’t generate instance Id! |
org.quartz.SchedulerException: Couldn’t get host name! [See nested exception: vcd1.vcloud.lab: vcd1.vcloud.lab]
at org.quartz.simpl.SimpleInstanceIdGenerator.generateInstanceId(
at org.quartz.impl.StdSchedulerFactory.instantiate(
at org.quartz.impl.StdSchedulerFactory.getScheduler(
at com.vmware.vcloud.scheduler.impl.QuartzSchedulerFactory.createInContext(
at com.vmware.vcloud.scheduler.impl.QuartzSchedulerFactory.createScheduler(
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)


Caused by: org.springframework.beans.factory.BeanDefinitionStoreException: Factory method [public static org.quartz.Scheduler com.vmware.vcloud.scheduler.impl.QuartzSchedulerFactory.createScheduler(com.vmware.vcloud.common.configuration.ConfigurationService) throws org.quartz.SchedulerException] threw exception; nested exception is java.lang.IllegalStateException: Cannot run without an instance id.
… 29 more
Caused by: java.lang.IllegalStateException: Cannot run without an instance id.
at org.quartz.impl.StdSchedulerFactory.instantiate(
at org.quartz.impl.StdSchedulerFactory.getScheduler(
at com.vmware.vcloud.scheduler.impl.QuartzSchedulerFactory.createInContext(
at com.vmware.vcloud.scheduler.impl.QuartzSchedulerFactory.createScheduler(
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
… 30 more

Well looks like we cannot get an instance id which prevents the service from starting up correctly. The reason for failing to get an instance id seems to be that we cannot resolve the hostname vcd1.vcloud.lab. A quick check with nslookup proofs that I obviously forgot to make the entry in the DNS server.

nslookup vcd1.vcloud.lab

** server can’t find vcd1.vcloud.lab: NXDOMAIN

After adding the hostname to the DNS server and a quick restart of the service my lab finally seems to be good to go for good now.


Categories: Homelab Tags:

VCAP-CIA lab preparations

July 14th, 2013 No comments

As I will be preparing the VCAP-CIA next and this exam, like the VCAP5-DCA, is performance based I will need access to a lab.

Section 1.11 of the blueprint covers the exam environment which is quite large compared to the VCAP5-DCA environment.

3x vCenter 5.1: 4GB = 12 GB
7x 5.1 Host: 3BB = 21 GB
4x vCD cell: 1GB = 4 GB
3x vSM: 1GB = 3 GB

Even if one host and vCenter are assumed to be the actual management cluster and can be ignored on building the lab this still totals out to 32 GB of RAM in total with possible swapping as the vCenter Servers in 5.1 will be on a lot of pressure on just 4 GB of RAM and everything else is also configured at the bare minimum. And this is without counting the additional software like Chargeback or the vMA.

As I do  not have access to shared storage yet (since I want to buy a proper NAS which is out of budget for now) I will also need to cope with the resources needed for a virtualized storage solution.

I intend to purchase an SSD for host swapping, so a little over commitment of resources should not hurt the performance too bad, but the actual host should at least have 32 GB of RAM. As I used laptops for the last couple of years I did not have a case and power supply lying around to re-use which meant that I would either need to buy a complete box or assemble it myself.

Getting parts in Ireland for a reasonable price actually seems to be quite a challenge, so I decided to buy a full blown box. I will add another 4 port NIC to the host and should be good to go, as I prefer to pay a little more and actually have someone else to build the box since I don’t have access to a proper working bench and tools anymore anyway.

I decided for an 8 core even if that would mean I would spend a little bit more money on the power bill but no matter what I will test in that small lab I don’t think I will ever be CPU capped again. I went for the 32 GB RAM even though non ECC, if one DIMM really turns out to be bad it will be cheap to be swapped out anyway. As for now I will use the OpenDedup Nasapp ( for shared storage within the virtualized environment to provide NFS storage but this might change as I still have a HP N40L with a 2 TB disk standing around in the office which I might turn into a NAS box until I can afford something better.

I opted against an environment build with autolab for this environment as it seems to be too far away from the actual test environment. Below you can find a picture of the build right now which I will add more VMs to when I start breaking down the objectives of the blueprint (adding 3 more vCD cells when covering objective 1.2 for example).


 The actual build of the lab was inspired by a blog post of Timo Sugliani ( I chose 6 hosts for now to have the possibility to test all allocation models while still being able to move machines around in a cluster and prepare or unprepare some hosts. For later objectives on the blueprint I will add another vCenter server, vShield Manager appliance and a couple of hosts to tackle the provider vDC and organization vDC management abilities requested in the blueprint.

Right now only local authentication is possible as there is no LDAP server running in the lab which is also a feature that will be added at a later stage when it becomes relevant.

This simple setup for now should still be able to cover most parts of the blueprint, so the next couple of posts will actually be starting to breakdown the objectives in an order which I personally deem least time consuming to learn and demonstrate.

Categories: Certification, Homelab, VCAP Tags:

VCP-IaaS exam experience

July 14th, 2013 1 comment

I sat the and passed the VMware Infrastructure as a Service Exam last Friday.

Gregg Robertson put together an amazing collection of study resources for this exam at:

I did not have a chance to sit the actual ICM course for vCloud Director 1.5 on which this exam is based but I personally used the following resources to study for the exam.

  1. – The official blueprint for the exam, was on version 2.4 when I took the exam but can be changed without further notice so this should be checked no matter what to see if anything regarding the objectives has been updated in the mean time.
  2. – Great place to start if you are familiar with the vSphere layer but don’t have a clue about how vCloud Director and other components interact with that layer. John Krueger does a great job explaining how everything fits together and shows how to actually click through a huge part of the blueprint.
  3. – Tired of installing everything from scratch when studying for a new exam? This one was a big help as lab time is important to get familiar with these “create, modify, configure” types of objectives in the blueprint.
  4. – vCloud Director Administration and User Guide being the most called out tool in the blueprint, followed by vShield and vCnter Chargeback should give you an idea on what to read in which order, map the “create, modify, configure” tasks from the blueprint to chapters in those guides and have clicked through them in the GUI at least 3 times should be a good preparation (at least that’s what I did).
  5. – Know your networking, the following posts from Duncan Epping helped me alot.
  6. – Alot of the blueprint has been covered on the vBrownBags as well.

The exam was a nice change to the VCAP exams (as you can actually mark questions for review and there are no impacts on performance as the exam is hosted in the test center), the questions were fair all the way through without any surprises outside of the blueprint.

Categories: Certification, VCP Tags: