Go has a precise garbage collector that knows where all the pointers are. So while yes, []byte is a (fat) pointer, the GC is free to move its target to a new location, as long as it updates all the pointers to it. This means that you may observe the address returned by &s[0] for some slice s changing even though you did nothing to s yourself.

The process of actually moving allocated memory around by the GC is known as compaction. It's not done aggressively today but Go reserves the right to do so in the future.