Yes! People can turn you off socially if you don't respect or belong to their "social culture".
Something as simple as a different hand shake can be interpreted in the wrong way.
Yes! It can appear like insignificant but your actions and social attitudes tell a lot about who you are.
Now, some people will enjoy diversity and connect with you precisely because you are different.
This will stimulate them.
Some others will disqualify you in a minute simply because your hair style or clothes does not match their culture.