In my experience, both are true. I'm more on the ML side, and I can tell I don't have the kind of routine and habits that good software engineers have, though I'm learning. But on the other hand, and I've seen this from software engineers who've made the transition to ML, and clearly have a good handle on the concepts (in one case even published papers in ML journals), they don't seem to have the intuition that allows them to select the right tool for the job, or understand when a given technique is appropriate; you could call that "modeling maturity" - a combination of mathematical maturity and the skill of relating domain knowledge to modeling choices.
Right, I see. That's not really possible imo. For things like mlops, sure. But model development, selection, evaluation? From what I've seen, it's exactly when engineers reach for standard tools without giving thought to how it applies to the given problem that they do a bad job.
Intuition is always important, but it shouldn't be the last word in an engineering problem. I think there is room for a lot more rigour in how we build, optimize, and validate ML model performance, so less of it gets left to intuition. The discipline is becoming mature enough that this is possible, I think there is a lot of room to build out "standards" and a "body of knowledge" that can be applied to building ML models. We're seeing it in pockets, but in so many cases, it is still a dark art.
And then from an actual software engineering perspective, so much ML code is just run-once jupyter notebook stuff... there is a lot we can do. I need to give this more thought, but I think it's acknowledged there is a big opportunity here
Even in non-ML software engineering you still have architectural tradeoffs, that while you can in part make reasoned arguments about, you still are relying on the intuition of your technical leadership.
Absolutely agree, the major challenge has been to explain to software engineers that the pick up an algo from github, monkey see monkey do a tutorial followed by automl or grid search is pretty irrelevant nowadays.
If you can’t implement any existing algorithm from a scientific paper it is pretty hard to understand the flexibility that is required designing a training pipeline.
In terms of tech stacks, cloud providers’ sample architectures and code samples are a base for cloud components and API understanding and not actual implementations.
