So far it's just a bit of smoke on the horizon, but I'm noticing some packages abandoning 'pip' installs entirely in favor of 'conda'. It's a bit early to tell if this trend will take off, but it does seem plausible.
Like, I understand it's great for managing package dependencies/setting up environments etc for a single project. But I've found it's an absolute nightmare for building docker images/generally doing build stuff when it's involved.
Not to mention conda installs seem to take waaaay longer than pip.
apt + pip works far more sensibly and reliably in my experience.
My experience has been the opposite: building Docker images is much easier with conda than it is with pip. With conda you can start from miniconda3, copy an environment.yml, and then conda create it. With pip, you might need to take additional steps to install system dependencies like build-essential first, and you'll need different tools to manage your virtual environment and Python installation too.
Yeah I know I'm on one side of the fence with this. The people I work with (not engineers) use conda all the time and love it.
A big thing for me is that apt + pip means I know more about what I'm installing, whereas conda seems like it'll go off and do what it thinks is best. If it's going into production then I want to know why we need package X and how it's been installed, rather than "conda says we need it".
Basically, I think conda encourages "install and forget" which means people don't really know or understand what they're putting into production. And that can cause a lot of problems further down the line.
Also the fact that conda installs everything to sandbox causes it's own issues. Suddenly I have two versions of a system package. Now I've got to do extra work to deal with that.
Then again, it could just be I never got over the time where uninstalling anaconda ripped apart the python install on my MacBook. That's a weekend I won't get back.
__
Also, Production shouldn't use virtual environments in my opinion. That's an additional deployment/build step which could fail one day. The container image is a virtual environment in and of itself anyway!
Sounds like you just need to train your colleagues to be a bit more disciplined with their package managing. It's not that hard to be clean about dependencies with conda. Maybe my take on it can inspire you here: https://haveagreatdata.com/posts/data-science-python-depende...
Read your post. Yeah, conda works fine with pip stuff. And yes, what you've detailed is similar to pinning versions in requirements. Which I do as standard -- I even do it with apt installs sometimes.
But I often need to install via system package management for other dependencies. conda doesn't respect the base system package manager. That is what causes headaches.
If conda respected system package management first, then installed as necessary, I wouldn't have a problem with it as an admin. But it doesn't because it's not built for engineering/admins (want stability + efficiency), it's built for scientific projects (want to run code easily).
Also, I'm using the "royal* we. Like, we as in admins generally. I'm the only admin in my team (voluntarily), so I need to be ruthless with this type of stuff.
EDIT:
I think you missed my point about virtual environments.
The entire container is a virtual environment. Why would we want to use another virtual environement for no reason except the fact that conda wants us to?
It adds extra steps which we'd have maintain. Which means more developer resource spent on maintenance. Which means less time spent on new features.
It's just another thing that could go wrong. Simpler systems break less often.
My use case is this: as a data scientist, I start new code bases all the time. Each project, simple experiment, data analysis, etc. needs its own cleanly separated dependency environment so I don't end up in dependency hell (I have 12 conda environments on my machine right now). Conda allows me to handle these environments with ease (one tool and a handful of commands -> as detailed in the article). With conda, I also have my data science Python cleanly separated from my system Python.
Of course there are other tools that can handle this use case. But pip alone won't do the trick. I don't like to have three separate tools for this (pip + venv + pyenv).
When I put something into production, I naturally want to keep using my conda environment.yml and have the same environment in dev and prod instead of switching to pip + requirements.txt, which might introduce inconsistencies.
I've built my project template around conda, primarilly so that a new user can just do `make install` and have all environments created. The docker portion also works fine, the only issue being that the images are perhaps a bit larger than I would like.
conda seems to spend a lot of time in a SAT solver or something to deal with the known NP-complete problem of dependency resolution [1]. Maybe pip doesn't do the same thing for requirements.txt setups?
It can be pretty bad, yes. A primary issue seems to be that conda package authors don't always do their deps right.
Pip, on the other hand, often "wins" by not even bothering to notice the versions of low-level libraries, etc. If that doesn't work, you get to keep both pieces.
Yeah, so while conda can be annoyingly slow, comparing it directly to pip in that department is unfair. A more fair comparison would be pip plus poetry or pipenv, which also have speed complaints :/
Conda does their own thing, so I hope that this doesn't become commonplace.
It's a nightmare of interop. Yes, it works for one person on a single laptop, but my experiences with conda outside of the happy path are universally terrible.
Just one from my personal experience: Conda installed Python having different compilation flags than official and distro Python builds, breaking ABI and causing native extensions built against official Python to crash when loaded in Conda Python.
A few times a year our office loses power, which causes the machine we have running for data evaluation to crash. There seems to be about a third chance for that to damage my Anaconda installation.
Last time I could only solve it by deinstalling everything and reinstalling an updated version. Took ages to get all packages running again. Luckily my code still works but now I get depreciation warnings (and not in my code). Who knows what will happen next time. It only took like a day I needed for actually working with the data as well... urgh.
As I understand it, conda has more sophisticated handling of native components, and is largely seen in scientific environments where Python is just the frontend to libraries that are actually in C++ and Fortran.
Conda is "pip" for people who know _nothing_ about pip/operating-systems/IT. It is basically a way to have a working environment without understanding that maybe pandas and numpy and scikit are not Python built-in modules, and that maybe you'd need a working C/C++ (and even a Fortran compiler too) to install it.
It's great for that purpose, but - as far as I know - it has not much to do with general Python dependency management.
I think this trend may be limited to fields relying heavily on natively compiled dependencies, which may not even have any python at all. Conda packages can be C++, Fortran, pretty much anything. That's the main draw for me -- otherwise, I'd stick with pip, which is definitely more mainstream.
