That's the difficulty when they bring the agent as the agents just tells you to send it to them. Just do your best to get the buyers info as well in case the buyer has any questions. Or if that doesn't work because the agent insists and the buyer doesn't realize, at least tell the buyer directly to feel free to contact you directly if they have any questions.
Anyway, jumping to conclusions here as not all agents will interfere. We were after a private sale last year and I was talking to an agent who we used in the past. He volunteered to go to the home and offer his thoughts with no expectation of a commission. We didn't bother because we were comfortable with our plan. Was there an ulterior motive? I don't know. Either just being helpful or he was trying to work towards selling our home. Either way the offer was appreciated.