Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Yeah Python got done dirty here. I tried a million tasks and I saw a peak of about 166MB. I'm still getting used to async in Python, but this seems to work fine though?

    import asyncio
    
    
    async def do_task():
        await asyncio.sleep(10)
    
    
    async def main(num_tasks: int):
        tasks = []
    
        for task_id in range(num_tasks):
            task = asyncio.create_task(do_task())
            tasks.append(task)
    
        print("spawned tasks")
        await asyncio.gather(*tasks)
    
    
    if __name__ == "__main__":
        asyncio.run(main(1_000_000))


Thank you! I fixed the code and updated the benchmarks. However, your 166 MB doesn't match my testing (I'm getting about 240M at 100k tasks) and 10x more at 1M. Are you sure you haven't misread your RSS?


Nope, that's what it is for me but you're using an older Python. 3.11 has a lot of improvements and not sure why you wouldn't use the latest?


If Python 3.11 is available, then asyncio.TaskGroup could be used instead of asyncio.gather() https://docs.python.org/3/library/asyncio-task.html#task-gro...


Running Ubuntu LTS and that was the version from the official repo. Will try a newer version then.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: