The simplest way to see this is that all parts on the outside surface of the conductor will remain at the same potential, irrespective of the location of the charge inside the hollow part.

Thus, if you think of solving first for the potential outside the conductor, and obtaining the field from the gradient of the potential, then the boundary conditions on the outside of the conductor remains unchanged by the motion of the charge inside. Because the boundary conditions are identical, the potentials will be identical, and thus the fields must be identical by the uniqueness theorem.