Prince Harry has been dealt a blow after the man who helped him make the Invictus Games a success quit.
