Open office plans were a reaction to teams that didn't communicate effectively (or at all). The idea was that putting everyone in the same area encouraged communication and cohesion. It also allowed for spaces that were chic and 'creative', because somehow creativity is impossible in offices that have the aesthetic appeal of a solitary confinement cell.
Honestly, I believe most of the value derived from open office plans is that it is harder to fake work and fake productivity. Now, everything that people were doing is out in the open and provides some measure of accountability. Maybe I'm being a bit too cynical, but I think that's a big contributor as to why open office environments are common.
And partially yes, it's because some people making the decisions don't understand the value of 'deep work' and the environments that encourage it.
While I'm sure there were elements of communications and observation in the widespread move towards open plan offices, I think it's very likely that real estate costs were at least an equally significant driver.
I had the pleasure to work in a genuine had-a-door office environment from 1997 through 2002. It was amazing for focus, productivity, and I found our employee communications to be generally excellent as well. I'm sure it was also stratospherically expensive on a per-dev basis, but probably still a rounding error relative to our compensation.
I hate open-plan offices. Deeply. I also see my overall "building occupancy charges" every month and am acutely aware of the pressures to be frugal. (I think real-estate is the wrong place to do it, but I understand the bean-counting view which is very apparent and productivity of expensive employees is not readily apparent [in a dollars/cents way].)
Honestly, I believe most of the value derived from open office plans is that it is harder to fake work and fake productivity. Now, everything that people were doing is out in the open and provides some measure of accountability. Maybe I'm being a bit too cynical, but I think that's a big contributor as to why open office environments are common.
And partially yes, it's because some people making the decisions don't understand the value of 'deep work' and the environments that encourage it.