I don't think you are insane but because others and your DW feels strange around him does not make him guilty of any thing or even deserving of suspicion.

Perhaps he is too friendly or too aloof or comes acrosss as distrusting because they have taken out his daughter, who knows?

Judge by what you know of him not delve into what if's.

I think it is totally unfair to a man who has very likely given a great deal to you during your lifetime so far. He also has had to deal with you living in a same sex relationship which can be tough on parents.

Apply the innocent until proven guilty model that our society has.