I agree with the previous posters... The chance is that you'll ruin a fine running thing, just because you would like to go where he does not want to go.
Yet, do not take this as the final verdict. His attitude, so far, has been supporting the view expressed above. But things do change. Most people are afraid and have little reason to trust other people. In all likelihood, he'll get used to you over the time. He will also start trusting you and that may, just may be your big opening.
Bottom line: do not do anything you may regret later; keep on doing what both of you like doing; and keep on watching the guy. Be ready to invite, if and when you feel that the time is right.
KD
