As soon as the device is removed, the box is no longer grounded.
The proper question is, why do the grounds and neutrals connect at the main service entrance, when they are separate everywhere else?
Neutrals carry current, grounds only carry current when there is a ground fault. That is why we separate them almost everywhere. We don't want a current path through conduits, water pipes, equipment frames, etc. If they are connected at, say, a receptacle, some of the current will use the ground as a path. Generated electricity takes all possible paths back to source, so we want the neutral to be the only possible path.
For the circuit breaker to operate under a ground fault condition, there has to be a circuit, of low enough resistance to have high current flow. That path is through the service neutral back to source. That is why we have to bond the grounds to the neutral at the main service entrance. If, say, the hot wire in an appliance shorts to the frame, the current will flow from the hot, to the frame, back through the ground, all the way back to the service entrance, back through the neutral to the transformer, through the windings and back to the service on the hot. This should be a low resistance path, resulting in high current. The breaker sees this high current, much higher than its rating, and trips.