You should tell him why this is bothering you, because you are inadvertently cutting off lines of communication by assuming you will create an issue.

If you don’t want to lose her forever, you are only postponing a difficult conversation. Sweeping things under a rug doesn’t resolve them