I don't know much about your relationship, but maybe he also worries that throwing in someone else (sex aside), might put a strain on your emotional relationship...jealousy issues, adding someone else's feelings into the mix...I find in a committed relationship a threesome only can work if both parties are extrememly comfortabel and safe with the other partner, that there are no insecurities or mistrust, best bet is to stop talking to us about it and start talking to him about all these things you are saying. =)