I've done the same thing with Pi 3Bs/4s basically for the same reasons. I definitely ran into the limitations of the SDCards and set up USB drives and NAS storage class for better performance. I ended up running my own docker registry on my NAS and running all of the images through that before deploying to the k3s cluster. I also hooked up container scanning and automated it the deployments through ansible.
Things I would do differently are using NixOS or bootable containers (CentOS) (side note, bootable NixOS container would be a killer app) and writing my own helm charts instead of fully customizing my manifests and doing the deployments from ansible, and would recommend against Raspberry Pis for the compute as the 3's and 4's don't support limits, e.g. cpu or ram limits, and I wasn't able to set up firecracker containers correctly on the Pis.
I'm also exploring hyperconvergence infrastructure (HCI) as that seems more like my ultimate goal for homelab stuff.
As far as Helm vs Ansible, I'm using Ansible to deploy the basics (bootstrap control plane & worker nodes, networks plugin) and then everything is deployed with IaC (Pulumi) which installs Helm releases.
I built mine before the 5 was released. I ended up running Ubuntu server on the nodes and configure them all using ansible playbooks (installing tailscale, k3s, updates, OS tweaks, etc.). I started looking at helm but there is so much inconsistency using community helm charts. I think writing my own would have been a better approach instead of templatizing my manifests and playbooks and doing it that way, however it is very easy to stand up a new service (assuming it's only 1-2 pods). If I end up DRY'ing my deployments it could end up being not too bad as a distinct deployment method from IaC or helm.
How do you like Pulumi? It seems similar to AWS CDK...
If you're templatizing manifests, kubespray does this pretty well I think. At least for the basics, it's pretty helpful so far. But indeed, I'm looking into deploying more things with Helm if possible.
Most services I've been using so far offer official Helm charts. But I get your point, it can be cumbersome and if there isn't an official one, then they can be pretty undocumented / hard to work around.
I haven't used CDK, but the concept is definitely similar. I think Pulumi most likely has wider support, since it's based on Terraform and even if you don't have a provider available on Pulumi you can "port it" (although never tried it, not sure if it works well). I like how it stores the state for you and secrets as well, saves quite a bit of trouble.
If you're looking at HCI Proxmox is amazing for homelabbing, with a 3 node cluster or 2 nodes with a qdevice. I use PBS for automated backups and RAIDZ and while there's quite a learning curve and initial setup time actual cluster maintainance is pretty hands off.
With enterprise gear it becomes outdated and then you replace it at your leisure, whereas with consumer equipment it dies so you need to replace it. The disvantage of that is the noise and power consumption so it's a tradeoff you need to consider.
PBS is the Proxmox Backup Server, basically it does differential backups of your VMs, integrity checks, and allows you to back up to multiple different servers. Proxmox does have a basic backup system that can save the VM images to disk though but I typically recommend the full PBS if you're building a cluster.
One interesting thing you can do is run PBS as a giant VM and back up or migrate the whole thing, just like you can run a small NAS as a VM as well.
Things I would do differently are using NixOS or bootable containers (CentOS) (side note, bootable NixOS container would be a killer app) and writing my own helm charts instead of fully customizing my manifests and doing the deployments from ansible, and would recommend against Raspberry Pis for the compute as the 3's and 4's don't support limits, e.g. cpu or ram limits, and I wasn't able to set up firecracker containers correctly on the Pis.
I'm also exploring hyperconvergence infrastructure (HCI) as that seems more like my ultimate goal for homelab stuff.