I did quite a lot of CUDA, algorithms, and parallel computing in general during my university time. I wrote my Master’s thesis on GPGPU. This was before it was applied to crypto mining or AI. I was supposed to stay at the Uni and do PhD in GPGPU, but eventually accepted a much better paid entry-level job at a company and, subsequently, I haven’t touched low-level programming since graduating and moving on to coding in Java/.Net/SQL, etc.
Over a decade has passed and you can imagine the level of my 20/20 FOMO frustration for not having carried on with the more advanced engineering stuff.
So my questions is: what resources and what direction would you recommend for someone like myself to narrow that gap of 10 years, such that I could hopefully apply for an interesting job in the field sometime 12 months from now? I am absolutely aware it would take a lot of studying, especially on AI/ML maths and algorithms.
Perhaps someone has already been in a similar situation and can share their experience?
I will much appreciate any input here!
An other point is that you don't need to write any CUDA code to be able to utilize GPU computing. If you need ML models, you have frameworks like PyTorch and Tensorflow. You just need to express your mathematical problem, and the framework will take care of the rest.
Even if you needed to write custom GPU code, you don't need to do it in C anymore! For example you can JIT Python, using Numba or Triton.
Usually writing custom code is only required when:
- You are doing something novel, like PhD level stuff
- You must optimize the ML project for performance and trough-put at interference time
- You need to brute-force solutions (be it crypto-hashes, passwords, NP-complete problems, ...)
My last point to you is that do you want to learn to use these pre-existing frameworks and libraries, or learn to develop them or maybe even create new ones? What ever your answer is, I'd say that the first option is a great stepping-stone to advance to the second one.