Ils avaient tout entendu is correct because it is a complete sentence that means 'They had heard everything'. Here, tout is the object representing what they heard; i.e., everything).
Ils avaient tous entendu would not be correct because it requires an object. It means 'They had all heard'. Heard what? The sentence is incomplete.
On the other hand, Ils avaient tous entendu ce bruit is correct, meaning 'They had all heard the noise'. Here, ce bruit is the object and gives the sentence full meaning.
Kwiziq community member (Learning French)