S'en aller

Hi, with s'en aller I get the meaning of an action of going away or just gone away, so quite different from the passé composé but,whereas the passé composé is very structured and always uses the same past participle for the verb the s'en aller expression seems to vary eg je m'en vais,tu t'en vas, giving the idea of a present tense action or an imperfect,IE I/you are going away/have just gone away BUT nous nous sommes est allé  and presumably vous vous étes est allé use the past participle of aller. So ,the question is,does this alter the meaning in any way and is it just a grammatical irregularity which has to be learned.?