> I challenge anyone to show a meaningful improvement that derived from telemetry of this kind
Which features of our software are people using, and where do we need to concentrate the most with our limited resources and bandwidth to provide our users the best possible experience.
What are the biggest build- and run-time errors, and are they a result of the developer experience, and can we fix this through either better documentation or better error messaging.
What are the paths to successful adoption of our software, and can we nudge potential users (through documentation or developer outreach) to get on one of those paths.
We polled our users and they said that they want feature A, but our usage data shows that a vast majority of our users need feature B to be more successful faster.
Our software is used most often with other-tool-A. Can we strike up an agreement with the team that makes A, in order to give our users a better experience when integrating both tools through documentation, tighter API integrations, etc.
> Which features of our software are people using, and where do we need to concentrate the most with our limited resources and bandwidth to provide our users the best possible experience.
The best possible experience starts with not being spied on, especially against my consent. No feature is worth that.
> What are the biggest build- and run-time errors, and are they a result of the developer experience, and can we fix this through either better documentation or better error messaging.
Make it easy to report bugs. This is a solved problem (c.f. any popular open source application).
> What are the paths to successful adoption of our software, and can we nudge potential users (through documentation or developer outreach) to get on one of those paths.
As a user, I never want to be "nudged". Ever. Do not "outreach" to me especially if you've made the decision to contact me by spying on my data.
> We polled our users and they said that they want feature A, but our usage data shows that a vast majority of our users need feature B to be more successful faster.
So your users told you what they wanted and you didn't give it to them and justified the decision based on analytics. This is where people say that analytics are good for the product manager's job in helping justify its existence but at the expense of the users.
> Our software is used most often with other-tool-A. Can we strike up an agreement with the team that makes A, in order to give our users a better experience when integrating both tools through documentation, tighter API integrations, etc.
Create great solutions. If that means doing biz dev or integrations to make that happen, go for it. You do not need to spy on your users to build meaningful partnerships.
> The best possible experience starts with not being spied on, especially against my consent. No feature is worth that.
This kind of knee-jerk reaction says that there is no comment in the universe that would make you entertain a different perspective. That's fine, that's not what this comment is going to be.
Instead, I'll say this: the default assumption of ie "collecting crash data equates to invading my personal privacy" is just wildly off-base. I will freely admit that there are plenty of companies who have abused the concept of the always-on-network connection and are doing hardcore data-mining almost at the keylogger level. That's bad, 100%.
But there is a large majority of data-driven companies who are actively trying to make your experience with their software better, with the small team that they have. To say "well f them, they can try to divine how I, Power Linux User, would even use their software" is reductive and puts an engineering team at a distinct disadvantage and slows their overall output. This is not what you, as a user of their software, want, and it is ultimately a self-defeating argument.
"Create great solutions" does not happen in the year of our lord 2023 without data. That data needs to come from somewhere, and "make it easy to report bugs" isn't it.
> there is no comment in the universe that would make you entertain a different perspective.
You're correct in that there is no comment in the universe that would make me ok with being spied on. That's not a "knee-jerk reaction" though, it's privacy and consent 101.
> "Create great solutions" does not happen in the year of our lord 2023 without data.
It's true that there's a trend toward dark patterns and spying, which is exactly why we as users should resist and not allow it to be framed as the status quo. Great software has been created for decades, a lot of it still in use, without a hint of spyware or analytics.
I've lived in the data-driven mindset for many years, and ultimately found that it still consists of mostly guesswork, opinions and strong biases everywhere. People make product decisions and they will seek the data to validate what they already want [1] anyway. It should complement, not replace talking to users, and is not a requirement to build great products.
When it comes to simple anonymized telemetry, one user or subset of users, spamming the same error over and over will skew your data, and unless you track them even more you won't be able to tell the difference. As I said earlier, it's a slippery slope to stand on.
Comprehensive telemetry/tracking is fine for say, an e-commerce website or any kind of cloud application where you're already receiving all the user input anyway. It's also fine, with consent, for desktop apps, especially complex ones like IDEs etc. It is not OK for third-party software packages that you'll ship to your own users - regardless of any promise to not add tracking to the resulting build, since at this point they already broke that wall and it only takes one "well-intentioned" PM to make it happen.
[1] a system that evolves automatically by running stochastic A/B tests is something many have dreamed up including myself. I'm really curious what that would end up looking like. That's being truly data-driven!
> Make it easy to report bugs. This is a solved problem (c.f. any popular open source application).
How can you make it easier to report a bug for the average user than providing a single button to the user that lets them upload a crash report after a crash?
Oh, man. Don't call user-initiated data collection "telemetry". You are going to lose that battle for no good reason if you insist on associating with automatic collection.
This comment is at the heart of so much that is wrong with modern software:
> Which features of our software are people using, and where do we need to concentrate the most with our limited resources and bandwidth to provide our users the best possible experience.
That a team of product managers and UXers can answer this question even half correctly with a bunch of metrics and A/B tests is one of the biggest fictions of our time (not to mention pure arrogance). Talk to your users, or better yet be serious users of your own product.
> What are the biggest build- and run-time errors, and are they a result of the developer experience, and can we fix this through either better documentation or better error messaging.
Invest enough in quality so that users see errors so rarely that they get in touch directly when they do.
> What are the paths to successful adoption of our software, and can we nudge potential users (through documentation or developer outreach) to get on one of those paths.
This is an anti-pattern. Your adoption is irrelevant, only the utility of your software and user happiness matter. If those things aren't compatible with you making money, you shouldn't make money. Using telemetry for this purpose decimates the already weak value of telemetry for improving your software.
If you are building applications with telemetry for this purpose, please stop.
> We polled our users and they said that they want feature A, but our usage data shows that a vast majority of our users need feature B to be more successful faster.
This is the height of arrogance. Quite incredible that anyone would think like this. I would immediately end the relationship with any provider who believes that their usage data, of all things, is enough to reliably decide that the users don't know what they want and need and the provider knows better.
> Our software is used most often with other-tool-A. Can we strike up an agreement with the team that makes A, in order to give our users a better experience when integrating both tools through documentation, tighter API integrations, etc.
The level of telemetry required to know not only how I use your software but what other tools I'm also using is quite terrifying. Pretty much total surveillance. Try talking to some users instead.
But I offer two interesting and related observations.
The first is that it's not that uncommon for truly terrible feature decisions to be made on the basis of telemetry. Things like removing features that turn out to be really important even though they are rarely used, etc.
The other is that I see some people who are using software that they know is collecting telemetry alter their use of the software in an attempt to influence decisions based on that telemetry. Things like using critical features more frequently than they otherwise would, in the hopes that the feature won't be cut.
Which features of our software are people using, and where do we need to concentrate the most with our limited resources and bandwidth to provide our users the best possible experience.
What are the biggest build- and run-time errors, and are they a result of the developer experience, and can we fix this through either better documentation or better error messaging.
What are the paths to successful adoption of our software, and can we nudge potential users (through documentation or developer outreach) to get on one of those paths.
We polled our users and they said that they want feature A, but our usage data shows that a vast majority of our users need feature B to be more successful faster.
Our software is used most often with other-tool-A. Can we strike up an agreement with the team that makes A, in order to give our users a better experience when integrating both tools through documentation, tighter API integrations, etc.