Operating systems:
I mostly use Arch Linux personally. For work, though, I think it's hard to beat everyone in the organization being on macOS.
Databases:
IMO, never use anything except for Postgres for OLTP needs. Also, you don't need a document store - you need Postgres (JSONB works well if you really need to store unstructured JSON). If you need to search, Elasticsearch/Opensearch just...works well. Never use MongoDB, ever. Even if it was reliable, easy to maintain and reason about, why do you need it?
If you need a lightweight local database with OLTP capabilities for managing application state, use sqlite3.
K/V store? Redis. Never had a problem using Redis at massive scale, just be smart about what you're putting there. Also works well as a message queue and pub/sub (Postgres actually works well here too). I've used Celery, ZeroMQ, and Kafka - they all work well too, no strong opinions about which one.
ETL/Data Processing/Workflow management:
Lots of hate for airflow in this thread. I don't think it's as bad as people are saying it is, but I moved to Dagster and am definitely never going back to Airflow. Dagster takes a few hours to grok, but then it just makes sense, and is a lot easier to reason about.
General data processing - a combination of polars, pyarrow, and ray. Store streamable source data in parquet, stream it to arrow while doing initial conversions. Memory map the files, process them in polars - distributed amongst ray workers, ideally abstracted through Dagster. Sometimes you have an existing data pipeline with some steps in spark. That's fine - orchestrate those tasks in dagster!
IDEs:
I've tried them all. Nothing beats the JetBrains stack with the IdeaVim plugin - I end up using almost all the tools - PyCharm, IntelliJ, GoLand, CLion, DataGrip, WebStorm, and a few times I've even used PHPStorm and MPS as well. I keep VSCode around for...text editing mostly. Most of my bash scripts and config management is done through vim (vim is also the only good Haskell IDE I've found).
> Nothing beats the JetBrains stack with the IdeaVim plugin
> I keep VSCode around for...text editing mostly.
Couldn't agree more. I would like an alternative to VSCode for text editing/viewing that has fast startup, basic syntax highlighting, tabs, and an auto-saving "unnamed files" feature, but a license for Sublime Text feels way overkill for what amounts to the equivalent of a throwaway notepad
Autosaving unnamed files is for some reason a feature not many tools implement. I think I've first seen it in Notepad++ and it just makes so much sense. The only thing I found that works like that without getting into sessions etc is Mousepad. It's also very fast to start. Don't know about highlighting. It probably doesn't exist. I would also give Kate a look. It's more capable than it looks at first and is my current note taking solution with a nested folder structure with md files. (It has recursive folder search and tree view of the folders. All I need)
I've been toying with JB fleet as an alt to VS code. It's certainly not on par yet, but for day to day light editing it's gotten good enough and has someil interesting differences.
The last few times I've launched VSC it's been way too many pop ups, notifications, dings, bells and wizzers that I almost forgot the task I opened it for.
Operating systems: I mostly use Arch Linux personally. For work, though, I think it's hard to beat everyone in the organization being on macOS.
Databases: IMO, never use anything except for Postgres for OLTP needs. Also, you don't need a document store - you need Postgres (JSONB works well if you really need to store unstructured JSON). If you need to search, Elasticsearch/Opensearch just...works well. Never use MongoDB, ever. Even if it was reliable, easy to maintain and reason about, why do you need it? If you need a lightweight local database with OLTP capabilities for managing application state, use sqlite3. K/V store? Redis. Never had a problem using Redis at massive scale, just be smart about what you're putting there. Also works well as a message queue and pub/sub (Postgres actually works well here too). I've used Celery, ZeroMQ, and Kafka - they all work well too, no strong opinions about which one.
ETL/Data Processing/Workflow management: Lots of hate for airflow in this thread. I don't think it's as bad as people are saying it is, but I moved to Dagster and am definitely never going back to Airflow. Dagster takes a few hours to grok, but then it just makes sense, and is a lot easier to reason about. General data processing - a combination of polars, pyarrow, and ray. Store streamable source data in parquet, stream it to arrow while doing initial conversions. Memory map the files, process them in polars - distributed amongst ray workers, ideally abstracted through Dagster. Sometimes you have an existing data pipeline with some steps in spark. That's fine - orchestrate those tasks in dagster!
IDEs: I've tried them all. Nothing beats the JetBrains stack with the IdeaVim plugin - I end up using almost all the tools - PyCharm, IntelliJ, GoLand, CLion, DataGrip, WebStorm, and a few times I've even used PHPStorm and MPS as well. I keep VSCode around for...text editing mostly. Most of my bash scripts and config management is done through vim (vim is also the only good Haskell IDE I've found).