The man who never lost a battle is losing to his own family