Our 30kloc Julia codebase only has one key macro, but it's a really important one. It's ~70 lines of code that's saved us ~5,000 loc, and more importantly let researchers do a lot of stuff that would need normally need more programming expertise. We might be able to live without it, but it would have cost us a significant amount of time maintaining and debugging, and a much slower research feedback loop.
Other languages with less powerful macro systems often create a DSL (domain specific language) for this kind of problem. With Javascript it can even be mixed in with normal code through the use of tagged template literals. For example:
const query = sql`SELECT * FROM User WHERE name=${name}`
It is so widely used in the JS ecosystem that editors often have plugins to add proper syntax highlighting to some tagged template literals and some frameworks pre-compute some tagged template literals at build/bundler-time.
It is definitely much harder to do it compared to Lisp-like macros though. But maybe the barrier to entry to create such powerful abstractions is a good thing. In my previous post I didn't say you should never use powerful abstractions, you should just not overuse them.
We did in fact start by creating a data structure that we parsed!
The main problem we were trying to solve is essentially that we had a lot of code like:
f(x) = sma(ewm(x))
g(x) = slope(ewm(x))
And we wanted to avoid recalculating ewm(x), while also not holding memory for it for longer than necessary. Our production use cases would have tens of thousands of intermediates, so the potential saving from getting this right is huge.
Naturally a DAG is a great fit for this type of problem. We started by creating an explicit DAG, where everything like ewm(x), sma(ewm(x)) would be a node in the DAG. We could then evaluate the DAG efficiently using multithreading, dropping nodes when everything that depended on them had been calculated, etc.
And I would usually stop there, but there were two main challenges. One was that the syntax was pretty clunky, and we wanted researchers (who weren't software engineers) to be able to use it. The other was that we wanted _any_ reference to something like ewm(x) to map to the node, even if they were used in very different places – and it was quite difficult to map these dependencies without explicitly tracking.
It's hard to quickly summarize our macro, but basically what it let us do was write the short syntax and track dependencies using the underlying Julia expressions. We mapped those expressions to nodes in the DAG, and that let us calculate the dependencies without having to track them explicitly. I don't think this would have been possible without some amount of homoiconicity and macros.
