A permeable rock breaks down quicker than a rock that is not so permeable because there is less space for anything (i.e. water) to get in the not so permeable rock in order to widen the holes and ultimately break the rock. On the other hand, a permeable rock is full of holes and will break down because of the many holes available to widen.
