Nested ESXi deployments are one of the best ways to test VMware products and their functionality in a lab setup. While I was setting up my 6.7 P01 environment to do some testing, I ran into a weird issue of ‘tryFormatDevice’ that i hadn’t seen before with nested installs.
The error is about the installer not being able to execute ‘tryFormatDevice’ command and failing after 3 retries. The exact error is as below with the text and screenshot reference.
Unable to successfully execute 'tryFormatDevice' after 3 retries 
Lot of articles online point to the issue of ESXi partition and mostly for ESXi 6.5 version. ESXi needs at least 16GB storage and sometimes the installer fails with an error same as the one in my case but for low storage space allocation on the boot partition.
In my case, I checked and confirmed that I had already assigned a 30GB disk to my VM. I had also turned on the hardware assisted virtualization feature on the VM that was running the ESXi.
I then ran into an article by Willam Lam about another issue about running the Nested ESXi VMs on vSAN storage. The error though, is about the installer not being able to format a VMFS volume.
It looked similar to the issue that I was facing so I tried the command that he has provided on his blog. Voila! it worked.
Here’s the command :
esxcli system settings advanced set -o /VSAN/FakeSCSIReservations -i 1I ran it on all my ESXi servers hosting the nexted ESXi VMs and all of them booted up without any issues.
I wrote this blog post because the error I saw was very different and for a different version. This might make it easier for someone searching it for v 6.7 and also for the ‘tryFormatDevice’ error.
Hope this helps! If it did, drop a message and let me know.