retourner vs rendre Passé Composé

Rendre and retourner have a slightly different meaning (in all tenses). 

Rendre is the standard verb to use if you return an object to its rightful owner.

Retourner has acquired an anglizised meaning to return something because it didn't suit or fit you or was faulty in some way.  In general, however, when used tranitively, it means "to turn something inside out".

J'ai rendu le livre à la bibliothèque. -- I returned the book to the library. You use rendre if that's the normal book return after you've read it.

J'ai retourné le livre au magasin. -- I returned the book to the store. This means that there was something wrong with the book or that for some other reason you wanted to give it back (and get your money back). Note, however, that this isn't the standard use of retourner.

Interestingly, the lesson on using avoir with retourner says

"Retourner never means to return (something) in the context of a shop for example" but the recent discussion suggests it’s an anglicism now in use.

Retourner can be used with avoir or être in compound tenses depending on its meaning in French (Le Passé Composé)

You're right. I did hear retourner used in the context of returning a flawed item, though. In general, however, it means to turn something inside out when used transitively. I amended my answer so as not to be misleading.

Retourner has several meanings :

With auxiliary ÊTRE in composed past

-to go back/ retourner. 

Elle est retournée vivre chez sa mère/She returned/went back to live with her mother

With auxiliary AVOIR

-He flipped a pancake by tossing it/Il a retourné une crêpe en la faisant sauter

-He turned the situation to his advantage/Il a retourné la situation à son avantage

INTRANSITIVE

-He was all upset to hear that/Il était tout retourné d'entendre ça