> The problem is that when a final binary is linked everything goes into it

I don't think that's the case on Linux, when using -gsplit-dwarf the debug info is put in separate files at the object file level, they are never linked into binaries.