I agree with Bonnie. If I were to suddenly say, "Let's go to yours", my friend would answer, "My what?" It has sounded very peculiar in other examples provided in various lessons, but I decided to overlook it. However, if someone is pointing it out and someone else is suggesting that it is normal, then I must say something. 

I could say, "Let's go to my house. On second thought, let's go to yours." But "yours" by itself is not specific enough. "Your place", "Your dad's place", "Your work", "Your playground", "Your school", etc. Listeners need that qualified.

I think I disagree with Don: people do say "let’s go to yours" or "are we going back to mine?", but it’s very colloquial. 

In the UK, "go past" and "go by" are commoner than "pass by"

Edit: sorry, I’ve another question. Does "Tu es repassé à la boulangerie"  imply you popped back into the bakery again rather than by, which to me means you went past and  didn’t go in.