I'd agree on the appearance alone - in most cases. Realistically though, there are also the really obvious ones.
Which is why I think you have to look at the totality of any given situation. Appearance, location/setting, behavior, context, etc. And usually, it isn't any one thing that sets people's hair on end. Nor do most people actively go over a checklist or observe for any long period of time. But when multiple things standout from the get-go, I can't fault anyone for erroring on the safe side.