Are you telling me he only knows it because of chest or is this just a joke and I'm over analysing this?

What I said in the description holds true, Rouge always keeps her girls on display so when they're covered up she may as well be wearing a mask. Also yes I'm implying that everyone she knows has always been talking to her tits and doesn't recognize her face.

Oh, I forgot to mention that this is specifically a joke based on issue 33 of the IDW comic, where Rouge goes by the false name "Facet" (like on a gemstone) so they can infiltrate chao races without her reputation getting in the way. The fact that her "disguise" doesn't involve any kind of mask leads me to believe Rouge's face isn't as well known as her reputation or...other attributes.