Elastic made the mistake of building a whole business on open source software by relying on the poor experience and artificial overhead to launching that software independently and in a production ready fashion. It's no surprise then that people in the community identify that this value is somewhat artificial. If you bridge that gap (either cloud providers providing it or someone contributes containerized/terraformed single deployment scripts), the value of having someone navigate the artificial complexity for you vanishes, and thus the value to the customer.
You have to question what value Elastic is offering its customers on top of the opensource project. Why is it that people complain about AWS devaluing the commercial services of Elastic but none of you complain about opensource devaluing the development of code in general. I can't for the life of me find a job that pays to write a new framework or some piece of interesting software because opensource completely devalues code. I think the proliferation of opensource is probably worse than some of you imagine. I'm not writing any opensource that isn't sponsored because I refuse to spend my time devaluing software development. The vast majority of the profits from opensource all end up in the hands of massive tech companies anyway. If we could skew the commercial advantages to the developer our industry would be a lot more pleasant.
Having given Elastic's support two tries at different companies, it doesn't surprise me that their business model is failing. Their support was _terrible_ both times; at no point were we ever in touch with anyone who seemed like they understood the product, cared about our issues, or were in any hurry to fix them. We were locked in year long, 6-figure support contracts in both cases, and issues dragged on for months until we basically gave up. We got better answers out of random Google searches and a 20 minute conversation with a friend of a friend.
AWS's hosted ElasticSearch only recently is able to handle the data set sizes we were dealing with, and their enterprise support on this (and other products) is vastly better than anything we ever got out of Elastic.
> We got better answers out of random Google searches and a 20 minute conversation with a friend of a friend.
On the other hand, if you were asking support to train you on "how to do A, B, C" while you didn't even check the documentation nor made a basic google search, I can understand why you were disappointed. Paying more for support doesn't change the nature of it, it doesn't magically become a google bot for you.
Could you maybe give an example on the issues you reported to support ?
If I have a six figure support contract and ask you how to do a simple thing with your product, you better have an answer. If it's so simple, why are you worried about it, when I'm paying for it?
We weren't having issues solved by basic documentation.
In the most recent example, we were occasionally hitting Java heap OutOfMemory under our workloads and wanted tuning or even architectural advice. It turns out that ElasticSearch didn't limit ingestion rate to control memory pressure and was happy to accept writes under load until it exploded. Heavy users of ElasticSearch commonly have to watch ES memory pressure and throttle their own writes client-side.
I would've loved to hear these limitations from elastic.co, be offered some tips on appropriate techniques for throttling, or have them accept a feature request to better handle this server side. We never got anywhere near that level of depth of understanding our problem, after months of trying. It felt like we were talking to first-level support who didn't understand the product much better than we did.
> I can't for the life of me find a job that pays to write a new framework or some piece of interesting software because opensource completely devalues code.
Could you put some more color around this part? I don't understand the argument you're making here. It sounds like you're arguing that there's not market demand for the thing you'd like to be paid to make because it already exists and that the version that exists is open source. How is that any different than if the thing you'd like to make is under a proprietary license but at a price point you are unable to compete with?
Disclosure: My entire career in the software industry has been at companies who pay people to write software that is made available under open source licenses. Whether or not there is demand for a new thing to be created has proven independent of the license on that thing. We create and support software that meets our customer's needs, give them a path to easily extending functionality with hiring a whole development team, and indemnify them from the risk of consuming potentially insecure, unsupported software in critical production environments. Open source is a software development model, not a business model. Providing customer value is a business model, and it's rather independent of the license underneath.
Being paid to write open source is the exception rather than the norm and very hard to break into even after decades of voluntary open source contributions.
> Providing customer value is a business model, and it's rather independent of the license underneath.
Yes, my argument and tl;dr is that flooding the market with free software has created an environment where superior for-profit alternatives (open or not) have no value to the majority, so software developers have got to go much much further to create a sense of value. There is of course intrinsic value to the software being used and it took skill and expertise to produce, it's just the perception of value is near zero due to the abundance of free alternatives and the culture of expectation that it should be free.
===
To support yourself financially when building a new library of some sort, you either require a commercial sponsor (someone who values this as part of a broader intent, thinks they will get kudos, free maintenance and hiring opportunities, but doesn't devalue their own business by open sourcing it), a derivative business from the software (support contracts, saas, book sales, bs enterprise features that should really have been there to begin with), or is run as a charity (donors or you simply just have to make $0 and lap up the praise for doing it out of the kindness of your heart).
The bottom line here is charity. People routinely work on software for no personal gain and people routinely expect all generic software to be free, to the point where a one off $5 per head charge is considered excessive and will never be considered.
If we imagined a world where people did place value on generic software, that is to say people expected to pay $1-$5 for a compiler or a library, that opens a lot of opportunity to individuals to compete and produce higher quality software and would give me the opportunity to work on things that I find interesting and care about.
Are you saying that most software engineering jobs in your experience are about integrating existing projects rather than building anything interesting?
I would say that all software engineering jobs are about integrating existing projects, although ideally that would be accomplished by building something interesting.
Often we draw some artificial line and ignore everything below that. If you draw the line right below where you're working, then you can "I'm building a system from scratch! Just using [Excel|a web browser|AWS services|NPM libraries|C standard lib|an assembler]."
I'm not sure there's any obviously "right" place to draw the line. The people working on better compilers are building on a stack of technology so vast they can't see the bottom, even if they're clearly many, many levels below where I am tinkering with some payment code talking to Stripe's APIs.
Not sure what the parent meant, but that sentence is most definitely true for the vast vast majority of software development of products - websites, apps (desktop, mobile etc..), IOT etc...
Making products is, as far as I know, also the vast majority of software engineering jobs.
I'm sure there are some software development work that is writing code that isn't integrating third party code- especially in library maintenance, low-level embedded stuff and some core business code of products.
Even if your products are novel and their core require specialized code, only a small number of the developers in the organization will work on that, the vast majority will work on things around the core.
I mean I was an embedded C dev, I "integrated" a compiler, an x86 chip, and a bunch of proprietary hardware. All software can be described as a kind of glue (luckily glue can be interesting lol)...
Some of us are systems/infra people for whom foundational stuff is where the interest lies. Some of us are product people who thrive on more direct user-facing value. It’s a lot easier for the latter group to find interest in integrating preexisting open-source building blocks, while the former wish they could be making the building blocks themselves.
But is that dynamic being driven by open source or does it just exist naturally? Like it doesn't make sense to code a complete OS from scratch every five years or write a brand new implementation of a web server. Code reuse is encourages by open source but it happens a lot with out it as well. And I'd argue it makes technology a lot more valuable because we have a suit of existing well built software that's been tunned over the years.
> Like it doesn't make sense to code a complete OS from scratch every five years
No but it does make sense for us to pay for using an OS given that it's fundamental to what we do. The free as in free beer model means that the entities that produce operating systems do not extract any significant fraction of the value they are responsible for generating. It also perpetuates this idea that we shouldn't pay for software at all, not even $5 for a compiler.
So with that in mind, I'm saying that we should be careful about what software we opensource, and ideally we should at least be commercially sponsored to produce the software that we do open source if we want careers in building these high quality components or if we want to breed a culture whereby employers do invest engineering hours building these things.
> And I'd argue it makes technology a lot more valuable
Yes it means that businesses are able to produce a lot more for a lot less, but it also means that my skills are valued less. Companies value engineers by their hours of input not by the commercial value of the code. For example, I think Electron is terrible for desktop app development. It's great for prototyping but it's bloated and inefficient, I think we can do a lot better. There is no real market for an alternative because we have QT on the high perf C++ side and we have the trash web code on the other. If we had a culture of expecting to pay small amounts for software it would be more likely that I could write a high quality middle ground based on Golang or something, which is less approachable than web dev but more approachable than C++, and we would have higher quality and better performing code produced by it. For us to have this thing I either have to build this thing for free also or I have to come up with some fairly contrived business model, hope that kind people donate etc. The backdrop of opensource culture has devalued this implementation beyond the point of viability, so we will not see it unless a commercial sponsor has a specific need for it or I'm feeling particularly charitable.
I disagree. I think because the incremental cost for another copy of some already written software is 0, the perceived value is 0. It's the general problem with copyright IMHO.
It's not like a widget, that there's a value intrinsic to each widget - the value is all in the creating the initial copy of the software and in any ongoing changes people want.
And while for a singular person, perhaps writing FLOSS is charity, but for companies it might well be more payment in kind - i.e. cooperation on making it do things each company wants makes it more valuable for everyone.
> Elastic made the mistake of building a whole business on open source software by relying on the poor experience and artificial overhead to launching that software independently and in a production ready fashion
It's a billion dollar business. I don't think they have made any mistakes, until now at least. They are very successful.
They are portraying themselves as a small indie dev up against a titan, but with over $400 million in revenue last year they are no small fish.
Yes they have been successful and continue to be. I still think they have made a mistake, I don't see why success would invalidate that. If amazon are eating into their profits their mistake was to not protect their business against that threat from the beginning. The mistake cost them whichever rewards amazon are reaping. That doesn't mean they have failed entirely, and clearly far from it.
> The mistake cost them whichever rewards amazon are reaping
I disagree because if they had had it closed or non-free open source from the start, they would not have a $15 billion business. Partly because people wouldn't have adopted it, and partly because it is build on other open source projects (Lucene).
I don't think there is a scenario where they get their cake (an open source project and a vastly successful business) and get to eat it too (they have no competition).
I think nothing makes this point better that their current business model (cloud hosting) is based on an acquisition. They had to acquire a business model...
I have a lot of empathy for Elastic here, it really sucks to see something you build being eaten by the 800 lb ruthless Amazon gorilla. But you can't have a company that starts with a bunch of OSS devs saying "open source lols" and somehow VCs throw money at it.
GPL2 doesn't prevent a closed fork of Linux. It requires that if you distribute your closed fork outside of your organization, you must also distribute the modified source under the GPL2 license. Large tech companies run their own forks of Linux without providing the source.
Linux had many reasons for its success, but overall governance was definitely one of them. People felt free to contribute meaningful code in the knowledge that the GPL would “defend it” against a future as free fodder for corporations. Corporations treated it with kid gloves because they were scared of the GPL, and this left the ecosystem free to breathe; and Microsoft’s all-out war on the “communist” licence effectively solidified a front of motivated contributors in a way that the existing BSD efforts had never managed to.
Exactly. For corporations, it was like the guaranteed mutual destruction which keeps peace stable. They felt safe to contribute because their competition could not steal the contribution from the communal pot.
A similar setup is needed for the SaaS age. AGPL is a good step but is problematic for corporations because of the patent grant clauses, the way GPL2 is not.
If open source weren't a thing, software development would be much bigger pain than it is now. If something goes wrong in the code you're using, you can look into the source and at the very least implement a hack if you can't change the base. All that goes out the window with open source if you throw it away.
Also, software developers would be much more rare since the cost of learning would've been much higher.
That doesn’t require open source, just the source. (Some?) Borland compilers used to come with the source of their standard library, which allowed us to hack around bugs in their time() implementation, but it wasn’t open source.
You have to question what value Elastic is offering its customers on top of the opensource project. Why is it that people complain about AWS devaluing the commercial services of Elastic but none of you complain about opensource devaluing the development of code in general. I can't for the life of me find a job that pays to write a new framework or some piece of interesting software because opensource completely devalues code. I think the proliferation of opensource is probably worse than some of you imagine. I'm not writing any opensource that isn't sponsored because I refuse to spend my time devaluing software development. The vast majority of the profits from opensource all end up in the hands of massive tech companies anyway. If we could skew the commercial advantages to the developer our industry would be a lot more pleasant.