> we need more languages like Dhaal or SQL or systemd, which are explicitly not Turing-complete.
Agreed 100%. First time I used Dhall a similar thought crossed my mind about it being an ideal specification language for most ETL pipelines and similar “business logic” work.
I feel like Spark was (in intent) an initial foray into this kind of solution with the data frames and SQL interface, but it falls short in number of ways and is way too attached to a single implementation (warts and all).
The idea of a “resource descriptor language” (4) is a very cool concept, and not one I’d thought of before. Would this be granular compute resources (x % of CPU and memory, a la Kubernetes resource limits) or more like terraform?
Maybe I’m misunderstanding, but isn’t (5) less of a language, and more some kind of constraint-solver based on (2) and (4). That is, it attempts to find the “layout” and allocation of resources for the requirements (optimise for minimum hardware, optimise for maximum throughput, etc). Again, this could be me misunderstanding this stage though.
Regarding (2), I think categorical database schemas might be an answer. But it still pretty much theoretical.
I am not really sure about (4). I think it would be a mix of both, although the actual Terraform operations would be treated as primitives.
I think (5) doesn't have to be a constraint solver, it might be just a language where you marry the resources (and the way they scale, also described in (4)) with the actual computation, even if it was all explicitly specified. (Constraint solving it is another level.) Basically, (5) describes the final architecture of the system, while (2) specifies what needs to be computed and (4) specifies what resources are available to do any computation.
Agreed 100%. First time I used Dhall a similar thought crossed my mind about it being an ideal specification language for most ETL pipelines and similar “business logic” work.
I feel like Spark was (in intent) an initial foray into this kind of solution with the data frames and SQL interface, but it falls short in number of ways and is way too attached to a single implementation (warts and all).
The idea of a “resource descriptor language” (4) is a very cool concept, and not one I’d thought of before. Would this be granular compute resources (x % of CPU and memory, a la Kubernetes resource limits) or more like terraform? Maybe I’m misunderstanding, but isn’t (5) less of a language, and more some kind of constraint-solver based on (2) and (4). That is, it attempts to find the “layout” and allocation of resources for the requirements (optimise for minimum hardware, optimise for maximum throughput, etc). Again, this could be me misunderstanding this stage though.