I know it's inefficient. But I'm speaking from experience where we've had incidents at both work office and condo building. It's been brought up as a concern at our Condo AGMs from residents. Whether it's food delivery or what have you, they just walk on through, when visitors should be buzzing up regardless. Break ins do happen.
My point is, if it's not someone you recognize, doesn't hurt to ask. If someone didn't recognize me and asked if I lived in my building, how difficult is it for me to just let them be on their way, and just tap myself in with a fob. Not difficult at all.
Why make something out of nothing. A Condo building is not a public place, it's a private community of homeowners and residents who need to all share in ensuring it's a safe and secure space. Concierge isn't at the desk every second of the day. They will do the same thing and ask people walking on through that they don't recognize, if they live in the building.