My phone doesn't fit in my jeans pocket comfortably either unless I'm wearing my side pocket joggers. The point is today's smartphone camera tech is still inferior to 1" sensor P&S's, which btw is smaller than a M43+lens. I have no problem putting my RX100 in a jacket or belt case, so it is the camera that is always on me.
If your phone is good enough for you, that's great. It's usually enthusiasts that are more serious about photography who buy premium P&S's these days anyway.
Smartphones are fine for today's disposable images shared on mobile social.