Locked out of 2 Passive Points

In act 3, you get the sacrificial heart to use at the ritual site for 2 points. I grabbed the heart, forgot to go use it, progressed through the rest of act 3, and now i have no way of getting to the past to go use it because the portal is now closed, and there are no waypoints i can use to get there.