Definitions of odd man out:

• noun:   someone regarded as eccentric or crazy and standing out from a group