Hi Loveli1,
I truly understand the issues and the confusion you are feeling. I maybe jumping on this bandwagon a little late, so please excuse me.
I am a recently single dad, with a four year old daughter. From a male perspective, it seems your 'guy' is after a comfort zone. Away from home friends, family etc. He appears to be looking to you for certainty & stabilty of some sort. The "I'm not the guy for you" is understandable to a degree, and if it was just a sexual experience for both of you; then OK. But the human heart is a fickle thing, and after desperately trying not to be interested in relationships again after being burnt so bad, I thought the "sex no strings attached" concept would be great.
I have only slept with one other person since being single again, and so quickly the heart kicks in and seems to endlessly confuse things again.
You're young, life is short, so live for you;- Carpe Diem!