He confronts you as you return to the house. You are supposed to have his permission before leaving