It goes a lot deeper than that. I've found this cropping up in relationship to the polyamory community, affecting what sort of spiritual work I do, and how I've approached MIT over the years.
I've come to believe the root of the issue may have to do with how I approach comfort. I totally understand there are people who seek similarity in order to find a space in which they can be comfortable being who they are. For myself, I'm going to be reasonably comfortable where-ever I am. I want my comfort to come from within, not from aspects of a space that I've chosen. I can learn from people who are different, so I do. I do tend to select for spaces that tolerate and encourage difference. For example I'd be very unlikely to go to an event that welcomed only blind people but much more welcome to an event that discussed issues related to blind people, welcoming anyone with an opinion on those issues. In fact I think the root of how I got started on this path does come from interactions with the blind community. My parents were very big on asking me to spend the effort to work on adapting the world to my needs and work on adapting myself to the world. They wanted me to be in a mainstream classroom in a public school rather than in classes with other blind people or going to the Texas School for the Blind. I found over time that I got to be fairly good at interacting with the world, particularly the world of technology. I tend to prefer regular computers and software to software specifically designed for blind use. I've been told a number of times by blind people that I'm unrealistic in what blind people can do with technology. The times I've asked organizations for help in the last 20 years have all resulted in me concluding that we think about problems differently, because I'm much more focused on adaptation within the rest of the world.
I'll share an example that shows how deep this runs for me. In our original conversation, diatom talked about a naked dance where in order to attend you needed to be naked. I had a strong negative reaction when I heard about this. It's not to the nudity; I will generally choose nudity in clothing optional spaces. It's more that in order to make people more comfortable, they will choose homogeneity; dancing naked in front of a bunch of naked people isn't as difficult as dancing naked in a mixed crowd. I totally respect people's right to have such a space. I even agree that as you're becoming comfortable with your body such a space can be incredibly useful. My dislike isn't strong enough that I would be unable to attend such an event. However, I'd rather be at an event that encouraged people to stretch their own boundaries and work to be comfortable from within.
Obviously this is not absolute. The IETF is more uniform than it should be, and I still attend. Even if the IETF fixes its diversity problem it will still be very much towards the geeky side of things. Also, while I prefer broad spiritual groups within the neo-pagan community, I do work within that community. It's all relative. I'm just pleased to have found an explanation for a number of behaviors and emotions within myself.