Game not allowing me to leave town

Act 2 Start of Buried Shrines. Went back to town for a few minutes. When I went to go back to map it won't let me travel. Can't travel to another location through waypoint nor desert map neither. I can travel through desert map, but it won't allow me to leave the caravan and my portal to buried shrines is still there. I hard stopped the game for 30 minutes and reloaded in and the issue still persists.