It may just be that he doesn't feel you're responding enough at the time, even though you tell him you want him to other times. I know it's no fun for me doing something if I don't get some positive feedback.