That is a good observation. In fact it is one of the possible endings for the universe known as the big crunch. The universe appears to be expanding and as gravity gets weaker the further apart objects one theory is that everything in the universe could drift further and further away with time, until it all dies and one by one the stars go out. For a black hole to suck anything in it usually needs to exert force on it to change its direction, if the object is travelling fast or is very far away this is hard for the black hole to do. On the other hand, if the gravitational attraction in the universe is stronger than whatever is driving the acceleration everything will be slowly drawn together and black holes will eventually eat everything up and then collide with each other, until all matter in the universe is in one point, this is the big crunch death of the universe.
It is a great question.
Fortunately most stuff is in orbit around black holes and won’t fall in any time soon. It is possible though, a very long time in the future. The universe is still quite young though, the Sun has only gone round the Milky Way 20 times since it was formed; and well only get round another 8 times before our nearest neighbour galaxy (Andromeda) will collide with the Milky Way.
Who knows, but trillions of trillions of years in the future there might only be black holes left.