Randomly added to a party

I was mapping by myself and when I portalled back to my hideout, I was suddenly in a party with another player. I didn't invite him, he says he didn't invite me. We're not in the same guild or any chat and have never previously had any contact with each other.

We were able to use party chat and leave the party normally.