Does negative sign in graviatational potential energy have any physical meaning or it is just due to mathematical considerations?

Off the surface of the Earth, there’s no obvious reference point from which to measure gravitational potential energy. Conventionally, we say that an object that is an infinite distance away from the Earth has zero gravitational potential energy with respect to the Earth. Because a negative amount of work is done to bring an object closer to the Earth, gravitational potential energy is always a negative number when using this reference point.