" ce pull ne va pas du tout. je le retourne tout de suite"

I thought retourner shouldnt be used in the context of returning something to the shop,,, I hear " Retourner never means to return (something) in the context of a shop for example. In French, we use rapporter (to bring back) or échanger [quelque chose]  can you say je le rentre? and or je le rapporte"? , 

Someone mentioned "I don't think retourner means to return in English like returning an item to a store (render quelque chose au magasin pour un remboursement) or putting an item away. is it the same for returning an item anywhere else than a shop like say a library? 

Another mentioned 99% of the time retourner isn't used to say you returned something in general. So is: "J'ai rendu les livres a la bibliotheque"  & j'ai rapporte les livres a la bibliotheque more preferred than j'ai retourne les livres a la bibliotheque. is it just the matter of choosing what sounds better in this case? even though you could possibly use retourner

& what is the difference between s'en retourner and retourner