Out in England, Harry was pictured wearing a palm tree embroidered cashmere cardigan from The Elder