Yes, take care of VM hw version. Also VMDK file size supported in 5.5 is 2TB+, hence vMotion from 5.5 to 5.1 will not be successful if VMDK file size of the VM is more than 2TB
However, It is always better to have on same ESXi version.
Please read: Section :Reapplying Licenses After Upgrading to ESXi 5.5 @ http://pubs.vmware.com/vsphere-55/topic/com.vmware.ICbase/PDF/vsphere-esxi-vcenter-server-552-upgrade-guide.pdf
VMware KB: Installing or upgrading to ESXi 5.5 best practices