Yeah, I built a library to run tasks based on a directed a-cyclical graph (DAG) and each task can optionally belong to a queue.
So I had to write a simple queue, but since I wanted demos to work in the browser it has a IndexedDB backend, and I wanted demos it to work in an Electron app, so there is a SQLite backend, and I’ll likely want a multi-user server based one so there is a Postgres backend.
And I wanted to use it for rate limiting, etc, so limiters were needed.
And then there is the graph stuff, and the task stuff.
There are a lot of wheels to-create actually, if you don’t want any dependencies.
I do have a branch that uses TypeBox to make and validate the input and output json schemas for the tasks, so may not be dependency free for the core eventually.
Not the grandparent, but Airflow is painfully slow and inefficient.
Our reinvented wheel using posgresql, rabbitmq and EC2 runners has ~10x better throughput and scales linearly with the number of pending tasks, whereas the airflow falls apart and fails to keep the runners fully occupied the moment you out any real load on it.
So I had to write a simple queue, but since I wanted demos to work in the browser it has a IndexedDB backend, and I wanted demos it to work in an Electron app, so there is a SQLite backend, and I’ll likely want a multi-user server based one so there is a Postgres backend.
And I wanted to use it for rate limiting, etc, so limiters were needed.
And then there is the graph stuff, and the task stuff.
There are a lot of wheels to-create actually, if you don’t want any dependencies.
I do have a branch that uses TypeBox to make and validate the input and output json schemas for the tasks, so may not be dependency free for the core eventually.