You are correct, it is an absurd and nonsensical idea. Especially from the material perspective. People will tell you that you need only XY amount of weight, which is just few percent of all the mass in solar system... but last time I checked, we are not grabbing random soil from Earth and putting it into a form of a satellite like a sandcastle. We need very specific materials put together in very specific way, where some of them are in very short supply to be wasted on Dyson swarm.

Of course, somebody will chime in and tell that advanced civilization could transmute any matter into any other matter. But if that would be possible, then it is completely negating any motivation to build Dyson swarm at a first place, because you can just transmute your toxic waste from local your local dump into antimatter and you have virtually endless amount of energy at your disposal.