Thank you for your insightful answers!
I have thought about some of the points, and I’ll try to address those I have some contentions with clearly and respectfully. If anyone is in the mood to argue about the topic, I’m totally up for it.
I might be a bit argumentative, but in the end, I’m doing this because I’m really curious about this issue, and would like to have a clearer picture on this topic. Please correct me whenever you feel like my comments are incorrect.
During my university years in statistics, the most important thing I have learned is that more data does not always mean better data. In theory, I can imagine, that actually continuously surveying the user’s behavior might give the developers valuable information, but the thing is that the more types of data you collect in parallel, the more likely that there will be random correlations.
When doing statistical inferences, like this is something that looks good when we first think about it, but I wonder if the people who try to use the mass data collection for telemetry with this purpose, have the proper understanding of statistics to be able to separate real data from noise, and to find real causal connections. For instance, doing A-B testing is kind of the minimum to make the statistics meaningful.
I don’t know if FOSS developers put in the effort to plan out these tests to make sure their conclusions are correct, or do they just look at the big bunch of data they get and try to make sense of it?
This seems like a valid point, but I guess this is something you only have to do once per installation, as an automated survey of the hardware. I get the point of things like this, I’m more worried / concerned about the effectiveness of data collection that is continuous throughout the use of the software.
I remember hearing about this story, very interesting point, especially how it did end up giving insight on something that the wasn’t really planned for. I personally wouldn’t want my OS to send automatic crash information to the distro maintainer, but I do see the value in easy, direct crash reporting.
I get it’s a joke, but it’s really misses the mark for the discussion I would like to have. I wholeheartedly agree that feedback is super important, there is no denying in that. And not every method of gathering feedback is telemetry. And not every method of gathering feedback is giving you the proper feedback you actually need. Staying with your analogy, if you want your jokes to get better, you don’t try to get feedback from your employees, whose paycheck depend on you, for example.
Thank you for reading my replies this far. Based on the current discussion, there is one thing I’m still looking for an answer for:
Can anyone give me a good example or a counterexample of continuous and repeated surveillance of the behavior of a massive chunk of the users resulted in valuable information for developing a software?
Did it work better than having a few select people try the software out in a controlled environment, with maybe a pre- and a post-trial interview?