People get married and move in to each others' apartments. I have never heard of a tenant being kicked out because of this. Unless it was in some sort of special designated non-profit housing or something.
I knew a woman who once lived in a nonprofit co-op for single women over 40 only. She said that all of her neighbours were the most miserable people she's ever known. Most were divorced multiple times, bitter, whiny, nosy, etc. She couldn't wait to get out of there.
Another thing you want to watch out for is even if you do bring your kids into an adult-oriented building, an upset landlord can be the least of your concerns. Your neighbours can make life miserable for you and your family.