i have been beating around this bush for eons and so far my best release strategy is custom build wordpress container where plugins and themes are built in.

The problem i was facing was mostly that almost all of the plugins used in "general", do not come with a testing architecture in any way. How are you going to handle a "rouge" plugin not closing its <div> tag correctly?

I used to have "hidden" control pages where i would render the most used elements with demo content which i could check against. But then you throw things like composer into the game and most of that goes out of the window.

After all these years, my best approach is still making a backup, hitting the upgrade button, and praying :/

Great question! Its one of the things where Im very curious to see if my plans hold up when they meet reality.

I plan to solve it my a mix of process and multiple environments:

0) Base assumption is that Im starting from a working site and everything that may break it is going to be a change to the working site.

1) Every live/prod site will have a test environment. Like test.example.com

2) (process) The customer that wants to change something will work in Test. Or me, if Im the one indtaling the plugins. Plugin(s) are installed/php changed and then verified to be working as intended in Test (manuals testing basically)

3) (process) Customer never works in prod.

4) (process) When the changes made in Test are working as expected, Ill then update the docker images used for prod with the (verified) working changes.

5) Deploy new images to prod in-actice site.

6) Switch active / in-active.

7) Verify updated live site works as intended. If not, switch back to previous active bringing the live site back online without errors and go back to step 2.

Its not going to be absolutely error free, nothing ever is, but the intention is to (a) minimize the number of errors by enforcing some process designed to promote quality[1], and (b) when something breaks it should be very fast to switch back to a working site, so error fixing dont need to be done in a rush because the (live) site is down.

1: Yes the process is old school compared to todays CI/CD pipelines and multiple deployments to prod each day, but that is not the goal nor the business needs. A stable working site is.

Edit: not sure I actually answered your question. A broken plugin (missing </div>) would never make it past step 2.

Have you checked out caprover? It's sort of an all-in-one app platform, self-hosted app installer w/ one click docker installs for WP, Analytics, etc.

No, but now I will. Thanks

Interesting, I have a similar goal but in the Drupal space. My partner and I have been building Drupal sites for clients for almost 15 years and we keep running into hosting issues for our clients at scale. Most of the "managed" options get extremely expensive very quickly, but they're also not managed from a content/modules perspective.

We're trying to do the best of both worlds and provide a managed (albeit opinionated) install that leverages our experience on the development side with performance and ease of use on the hosting side. Would be happy to compare notes as it comes together. Our website is in my profile and our email is hello@ the domain listed there.

Interesting. I have zero experience with Drupal, but I assume its similar to WP in its architecture since you bring it up :)

I think any good solution to working with these 50/50 code and configuration mgmt technologies have to start with a solid analysis and understanding of what the actual business needs are(?). And then (re)evaluate how things are done.

And then perhaps also pricing models when it comes to the ‘“managed”’ part you mention. I think, for what Im building, there will end up needing to be a “managed hosting” offering which is more restricted and a custom development/T&M work which will be hourly.

Could be interesting to compare notes when get more of the setup up and running

I've been wanting to pull the ripcord on a marketplace platform, think a shopify for WP Maintenance services and freelancers, or an unbundling of fiverr's WP gigs marketplace. Basically some sort of service that merges tools like MainWP with dashboards, issues, kanban boards, feedback for managing clients, logins, hosting issues, etc. It just always seems to broad and scope to get started, and I personally kinda hate WP and would probably do it all in Laravel. I'd love to figure out how to do something like WP Maintenance but for legacy Laravel Apps, but I'm not sure there's a market fit for that.

hey I've been tinkering around to build something on the same lines using similar stack. Let's connect? I'm hey[at]bilal[dot]dev

> I’ve been a SW dev for 10+ years now and know very little about WP

WP has become a gigantic ecosystem with its own sub-ecosystems and specializations that pertain to those ecosystems at this point. Seeing it as a 'Linux ecosystem, but as a web app' would be a better way to understand the landscape. Therefore, you should definitely try to learn more about WP because:

> But it has always seemed crazy to me how everyone in the “WP space” seems to be making changes directly in prod o_O

...even starting from the need to understand that, there are a lot of things to understand. People make changes directly to prod because making WP high available is very difficult due to the HA-enabled infra that is out today not supporting stateful workloads well, hence its difficult to have dev, staging, prod deployments etc. Especially because a change in plugin code or setting (an update, changing a setting etc) may trigger database changes and one wont work without the other.

And when you end up setting up one, it comes out way too expensive for any individual or small business to afford paying for it. As opposed to a managed, non-HA WP hosting provider allowing you to host a WP site that handles million or more visitors every month for $45 as opposed to having to pay $200-300 for a HA setup so that they wont make changes in production and they will have copies of their site running. Which, they can also do by buying another $45/month hosting and duplicating their site behind an IP-gating etc by the way.

WP site owners use plugins to make happen what they want to make happen, and a lot of these plugins need to be able to modify files in the filesystem to be able to work their magic. This ranges from being able to resize thumbnails to creating sitemaps to more complicated ecommerce application needs. Some need to modify .htaccess in order for a major feature to work. Others may need to modify the config file to add something etc.

And at that moment the whole containerization, docker, deployments, multiple environments thing hit the wall: If all the plugins are built into the container, they wont be able to modify the filesystem, especially their own files when needed. Which is a game breaker for any average WP site owner.

You could use an NFS to mount the wp-content folder, and that would fix a LOT of your issues. However, cheap, reliable managed NFS offerings are extremely scarce. And what's there is expensive. So you will end up having to rack up the price to a point where most people wont want to pay.

Even when you solve that, you still have problems - you are now building the WP core into the containers, and therefore are able to deploy automatic WP version updates and also having improved performance, great. But what will happen when you deploy a WP version update that breaks various important plugins because they are not compatible with that version yet? Boom. Upset users. A broken plugin and feature is most of the time is as bad as your entire site being broken.

You could prevent users from installing plugins and you could offer a selection of plugins that the users can use. Assuming that you solved the NFS problem. Then you could avoid version conflicts, compatibility issues. But then many users wont use your service because they cant install the plugins they want. You could have a market like that and a lot of web agencies do that apparently, but they are for their clients with specific needs who wouldnt need anything more freeform and complicated like most of the WP users do.

...

The best fit on the market for solving all of these is Google Kubernetes Engine + Google Filestore + Google Cloud SQL. (people complain about AWS nfs offering having spotty performance - bad for WP). But Google egress charges literally kill any chances of Google having any foothold in the WP ecosystem because with their outrageous egress pricing, a small WP site owner that hosts a middling WP site for $25/month would have to pay another $15/month for egress traffic costs on average, and that makes google double or triple more expensive for out-of-the-box WP compatible web hosts right from the start. And that's a variable price too - no WP site owner would want to pay a surprise $100 in a random month because their site has had more traffic suddenly. Standard WP hosts can provide everything from cpu/db to traffic costs even for a million visitors for $50/month fixed.

So anyone who builds any WP SaaS or consumer grade hosting at Google infra would have to start with 3-4 times more expensive compared to all other offerings when they add their own profit margin.

If you end up choosing to not allow users install plugins and themes, using WP multisite could be much better for your setup - it acts as a singular WP installation that can host an infinite amount of WP sites inside itself, with various plugin bundles activated/deactivated for different packages. You can build this into a container, and if you don't have any plugin that needs to use the filesystem, you can even deploy it in K8 in front of a managed DB and with the media uploads sent to some S3 compatible storage or something. (That requires custom code).

Also - nginx is not a good choice for WP. A large swath of WP plugins require .htaccess to be present and accessible for adding custom rules for their features, and this makes Apache the better candidate for such a situation. Apache with event mpm and php-fpm behind it is as performant as nginx these days, so you can just slap Apache and not ever have to deal with the rewrite rules that your customers will be asking you to add to nginx rewrites.

At a first read through there seems to be a of usefull knowledge and experience in your post. So I’ll surely go through it in detail and take away everything I can.

That being said, unfortunately you have a few assumptions, that makes some of what you write less relevant.

> But then many users wont use your service because they cant install the plugins they want.

This is sort of the core misunderstanding: This setup is defined for one customer (marketing agency) so I dont really care how the masses would like to use WP hosting. And in extension of this users will not be allowed to freely install plugins, customizations etc. That is want it means for this to be “fully managed” (I wrote about what and how in another comment above). The business need for this thing came about in the first place because the customer didnt not want to be the one to fiddle with webservers, plugins and updates. So the goal is to build something that strikes a balance between enabling the customer to deliver their (marketing stuff) to their clients and doing it in a controlled manner, for an acceptable price. Restrictions and processes are going to drive prices down, while the need for flexibility and time-to-deliver will drive price up. The question is if there exists a balance thats acceptable to all parties.

On a more overall level, I’ll add that I get how WP has become a huge ecosystem like you say. But part of why I fint this little project interesting is that, in my eyes, its an ecosystem that has evolved through thousands of low-tech users (certainly not experienced devs) hacking together bad, but somewhat functioning, solutions and making so many bad tech decisions along the way, resulting in a mountain of tech debt which leads to all of the stacked problems you describe in your comment. I think you are completely correct in all the issues you describe, but Im interested in whats possible to do if all the usual ways of doing things in this ecosystem is thrown out the window, and we look at how things should be done. Is it possible that a better solution doesnt exist? Sure! But wont know still we try.

There is just so many ingrown “bad things” that seems to be the norm in this WP world, that would not even be considered in a more regular sw development situation: Imagine giving the somewhat “tech savy” end user access to the C# codebase, installing nuget packages (WP: plugins), making code changes (WP: functions.php) and messing with appsettings (WP: configs).. and directly in prod. Its just too far out. I wont even let the senior devs on my team deploy code changes directly to prod. We have git, PRs, reviews, tests, build pipelines, test environments and so on.. We’re not savages. Or, a less dramatic example: a few comments/people have asked now, what I’ll do about “that broken plugin which breaks the site”. Well, I’m not going to use it - its broken!? Just like I’m not using broken python packages.

> This is sort of the core misunderstanding: This setup is defined for one customer (marketing agency) so I dont really care how the masses would like to use WP hosting

In case you missed, I already made an example of that case. It is a workable format in the WP ecosystem and there are many who use it. However, the market for it is limited to the customers whose needs fit the specific features you provide. That said, its definitely possible to establish a small business with that format. But its naturally limited, and in the higher tiers the competition is fierce.

> I’ll add that I get how WP has become a huge ecosystem like you say. But part of why I fint this little project interesting is that, in my eyes, its an ecosystem that has evolved through thousands of low-tech users (certainly not experienced devs) hacking together bad, but somewhat functioning, solutions and making so many bad tech decisions along the way

That's the perspective mistake what a lot of people who are new into WP make. I thought similarly way back.

However as you get immersed in the ecosystem, first you see that things work quite well and they are not merely 'somewhat functioning', and the tech choices made by everyone exactly fit their budget, their needs and what's maintainable. People end up making the mistake of providing 'better solutions' that will surely help the 'bad tech choices' in a given use case a few times. Only swiftly to learn that there were reasons why that particular use-case and its sub-ecosystem developed, and those reasons almost 100% coincide with the business needs of that sub-ecosystem.

Case in point:

> mountain of tech debt which leads to all of the stacked problems you describe in your comment

Being filesystem-dependent never was tech debt. At the point WP and its ecosystem came to being, there was no availability of reliable NFS setups, leave aside anything being affordable for the broad public in any way. Coupled with the plugins needing filesystem features to provide features and enhance the sites, being stateful was a good choice.

Today its little different - the majority of WP ecosystem uses plugins and thats why WP is big - anyone needing to any kind of thing can find enough plugins to make it happen. Hence the need for the filesystem.

Like you are planning, its possible to identify a narrow set of feature needs and compile a list of plugins and themes to create a rather rigid version of WP and even turn it stateful. But this is a sub-ecosystem of WP and WP would never ever become what it is if it was this limited from the start. In the end, the plugins that you are going to compile into your WP distro will be plugins that were made for use in stateful sites to start with.

So in a sense, this is not WP's tech debt - its the Internet technology's tech debt - it was built on stateful apps, and those who were running stateless workloads were few at the dawn of the Internet. When the stateful workload problem is solved (either by K8 or by some other tech) this problem will be auto-fixed.

> Imagine giving the somewhat “tech savy” end user access to the C# codebase, installing nuget packages (WP: plugins), making code changes (WP: functions.php) and messing with appsettings (WP: configs).. and directly in prod. Its just too far out

That is why no more 'properly done' and 'well engineered' cmses, platforms have never taken off: The flower shop owner somewhere in Oregon or the Japanese blogger in Tokyo needs specific features for his or her website/shop. These are provided by plugins that immediately implement those features to his site at ridiculously low cost and effort. As his or her business scales, things will keep the same rhytm, with solutions that require major monetary and time investment for 'more properly engineered' solutions demanding insignificant investment of money and time from him. Over time, he or she will end up being proficient in this easy-to-use system to be able to run his WP-based site/shop single handedly, whereas similar setups require considerable money and even teams to set up and maintain with 'properly engineered' solutions. Naturally, he will never go near any such solution.

> We’re not savages.

Thats a very crappy way to look at this. The flower shop owner above will not like that kind of perspective. Neither the rest of the WP ecosystem. Such attitude and arrogance is not well-received. Especially keep it away from any support/comment thread at WP.org. You would get very harshly rebuked.

> Or, a less dramatic example: a few comments/people have asked now, what I’ll do about “that broken plugin which breaks the site”. Well, I’m not going to use it

Your users would eventually force you to make it happen, or they would just move somewhere else. If youre lucky, some of them would even generate enough noise and feedback for you to notice so that you could address the situation. Otherwise the majority just silently moves on to somewhere else since there are a plethora of WP services for anything. This is before the fact that there are plugins that you have to use even if it breaks some site. Its the providers' obligation to fix it and make things happen.

WP is all about making things happen fast and with reasonable cost. In between a service that does things right 98% of the time much easier and with lower cost and one that does everything right and 'properly engineered' 100% of the time, the users will go with the 98% one. The other one won't even enter the picture to be considered. The hardcore reality of real life economics is much different than the reality of the engineering in large organizations in which engineering can be done in specific ways without being able to demonstrate total business justification, thanks to being shielded from the market economics due to the large capital and complexity of such organizations being a wall that separates them.

Large marketing clients are much more workable in that regard and that's a good choice. But those large marketing clients will also have demands for features like the general WP users, even if the demand frequency may be much lower than the average. However make no mistake - even if they are larger users/customers who may have been fed up with maintaining sites and plugins, they are still WP users.

...

Your approach and attitude reminds me of mine way back. WP ecosystem taught me a lot about the real world, actual business needs, what really matters and changed my entire perspective. Especially interacting with customers/users teaches an engineer like nothing else could. So it will be a maturing journey however you look at it.

Well, we have a lot to disagree on. I’ll find the time to read your comment in detail and comment on some og your points, b/c I find the debate interesting. But I just want to say this right away:

> > We’re not savages. Thats a very crappy way to look at this. The flower shop owner above will not like that kind of perspective. Neither the rest of the WP ecosystem. Such attitude and arrogance is not well-received. Especially keep it away from any support/comment thread at WP.org. You would get very harshly rebuked.

It was a joke and just intended for the quote to add some humor to the post. I’m sorry if it didnt come across like that.

> I’ll find the time to read your comment in detail and comment on some og your points, b/c I find the debate interesting

Do feel free to do so. I've been dealing with this kind of thing (HA + scalability and bringing WP into that realm) for the last ~1.5 years for an important project, so Im pretty loaded on that front.

> It was a joke and just intended for the quote to add some humor to the post. I’m sorry if it didnt come across like that

Gah. Great.