Have you checked to see that you have the right firewall ports open? (particularly UDP)
How high is your vm consolidation ratio? what type of bandwidth and latency do you have between your client systems and the vm hosts?
Note that storage latency has a big impact on users perception of performance, try to aim for 1ms or less if you can.