I think our relationship is at a crossroad and I think we are on different paths. Should I end it?

We've been together 4yrs. I have 2 kids. He takes care of his sick parents. We kept the kids separate from the relationship for the first 2 yrs. He said he wanted to get in my world and spends every weekend with the kids and I. But, every Monday I feel horrible, I go back to my life and he goes back to his parents house. I know he loves me, but in the back of my mind wonder if he's just taking a break from his life to come into mine... I feel like its a relationship of conveince. He doesn't want to get married. He doesn't want kids of his own. I only want more. Yet feel guilty. We get along great. If I ended the relationship because I only wanted more after 4yrs, would that be selfish?