Ideally try to keep 20-30% slack space to avoid constant re-balance. The workaround for this is use the "reserve capacity" option on any VMDK (like say SQL) that you do not want to ever re-balance.
Rather than try to say what all VSAN can do (I've seen it do a lot!) its easier to state which use cases I wouldn't recommend.
- High streaming writes. The caching system doesn't have a cut through yet so while all flash will mitigate this somewhat, I wouldn't deploy your biggest, baddest Hadoop workloads on VSAN.
- Also if you need a PB of cold storage for video you will likely find that modular storage is a lot more cost effective than adding hosts just to add disks.
- I think ~150K IOPS is the most I've seen on a single all flash host on 6.0 so if you NEED more than that today you may need to go look at another more exotic solution.
Keep in mind that a host using VSAN can still use FC/NFS/iSCSI/FCoTR/IB/SAS/NVMe for workloads that didn't quite fit into VSAN for some reason. (Its not an all or nothing thing here). I had a customer need full disk encryption and didn't want to use Hytrust (it would have met compliance), so they ran a FC array for the DB's that needed encryption and ran VSAN for everything else.