With this in mind, it is possible to imagine a whole-
program transformation that would add locking on every
object manipulated in RPython by the interpreter. This
would end up in a situation similar to Jython. However,
it would not automatically solve the issue of deadlocks,
which is avoided in the case of Jython by careful manual
placement of the locks. (In fact, being deadlock-free is
a global program property that cannot be automatically
ensured or verified; any change to Jython can in theory
break this property, and thus introduce subtle deadlocks.
The same applies to non-atomicity.)
This is just not true. It is true for the general case, but not for this specific case. Remember the pithy solution to the dining philosophers (number all the chopsticks and pick up the lowest numbered one first)?
I don't know the details of the pypy interpreter, but it does seem that given a byte-code that affects several objects, you could just acquire the locks for the objects as ordered by the objects address.
Now the pypy guys are very smart and this is a simple solution, so I'm sure it yields horrible performance, but the fact of the matter is it's not true that "being deadlock-free is a global program property that cannot be automatically ensured" (it is true that it can't be verified, but if you generate all the locking code it can be ensured by strict ordering) and to say otherwise is wrong.
You missed this point: and moreover if you need to have accessed the first object before you can decide that you will need to access the second object
That is, he's talking about situations where they don't have full knowledge when entering a critical section which objects will be accessed. Hence, they can't order lock acquisition in such a way to avoid deadlock.
You should be able to determine the set of objects which might be touched by a section of bytecode such that they need locking. However, the majority of the time that set will vastly over-estimate the number of objects which actually get touched (the problem bytecodes here are, just from inspection, JUMP_IF_{TRUE,FALSE}, EXEC_STMT, IMPORT_STAR, BUILD_CLASS, {STORE,DELETE}_GLOBAL, probably a few more I can't be bothered to think through right now), and a lot of the time will simply be "all the objects in the system."
Fundamentally, it doesn't make a difference whether you analyse the python or the bytecode; further, I'm not even sure that RPython is ever converted to bytecode, so it's not entirely relevant here.
I don't know the details of the pypy interpreter, but it does seem that given a byte-code that affects several objects, you could just acquire the locks for the objects as ordered by the objects address.
Now the pypy guys are very smart and this is a simple solution, so I'm sure it yields horrible performance, but the fact of the matter is it's not true that "being deadlock-free is a global program property that cannot be automatically ensured" (it is true that it can't be verified, but if you generate all the locking code it can be ensured by strict ordering) and to say otherwise is wrong.