From my observations in pretty much any unoptimized Python (CPython interpreted) code function calls is nearly always a bottleneck. And speed is directly bound by the number of function calls being performed, not by ponderous data structures.
The ponderousness of these data structures isn't just about memory consumption or having to use boxed numbers. As far as performance goes, PyObjects infect everything in the interpreter. For example, when you're calling a Python function, after a long run-around in ceval, PyObject_Call, the function object's function_call method, you'll finally get back to ceval which creates a frame via a lengthy call to PyFrame_New. The whole process is a mess of allocating, deconstructing, increfing, decrefing, and tag-checking.
From my observations in pretty much any unoptimized Python (CPython interpreted) code function calls is nearly always a bottleneck. And speed is directly bound by the number of function calls being performed, not by ponderous data structures.