I semi-frequently send audio files to one of the aforementioned friends of mine, and that’s about it. Everything else is just images (which signal compresses to oblivion btw) and stickers.
The fact that there’s no way to view individual files that have been sent to/from me (+ their sizes) and remove them—like there is for messages.app in macOS—makes troubleshooting even harder here.
I will admit though that I haven’t looked too far into the problem but it’s seems to be an issue that just about every signal user I know deals with.