Look at the weight of some cars for comparison. Modern EVs aren't too far off as you say when compared with comparable performance cars:
Tesla model 3: 3920lbs (SR) 4056lbs (LR)
Kia ev6: 4017-4255lbs
Average new US vehicle: 4100lbs
BMW M3: 3,840-3,990 lbs
BMW M5: ~5300lbs
Audi A5: 3,682-3,990 lbs
Toyota Camry: ~3500lbs
Toyota Prius: 3,097-3,340lbs
Dodge charger: 5100-5900lbs
Yes, those BMWs are insane weights for an "M" ("Motorsport") series.
Especially considering that it's common knowledge in motorsports (although evidently less common than it should be) that more power only increases acceleration performance, but lighter weight increases performance of acceleration, braking, cornering, and tyre wear! Plus, reducing weight is compounding; take out one chunk of mass, and other components can also be made lighter, e.g., reduce the body mass, and you can fit lighter suspension components, brakes, wheels, tires, etc., and the light-weighting compounds.