Understanding and evaluating the behavior of technical users. A study of developer interaction at StackOverflow

As SO is an example of Q&A discussion forum, where anyone can participate and answer technical questions, we start the discussion by checking the spatial characteristics of the crowd. In other words, we check the result and the behavior of people according to their ‘location’. Specifically, we cross reference the questions asked in one country and answers provided by another. In our dataset, we took USA, UK, France, Germany, India, Russia, Canada, China, Brazil, Netherlands, Italy, Sweden (They rank high in terms of quantity of posts). The result for this test is shown in Table 1. During analysis, we observed that people preferred answering more to a person with the same nationality. Though, questions asked at SO does not have to do anything with geography, the pattern we observed clearly has a skew towards the parent country. This was in contrast to the initial belief, where we expected that people at SO participate in a nation independent manner. Therefore, after getting this result, we repeated the test 10 more times by taking different number of subjects. For instance, we tested with 500, 1000, 1500, 2000 and so on users from each country. However, the result was the same. The numbers were always in the favor of the parent country. Taking inspiration from these results, we subsequently took the cities from the same country and repeated the procedure. We chose USA for the purpose. It was chosen because SO is based in USA, and has the highest participation by numbers. In the experiment, we took New York, LA, Chicago, Baltimore, Houston, Indianapolis, New Jersey. Repeating the procedure, and cross-referencing the questions with answers, we obtained the same result. The crowd preferred answering to a person from the same city. We dug more deep and repeated the test for different states in the country. We took Virginia, Texas, California, Florida, Pennsylvania. However, during analysis the result was the same (preference of people towards the same state). Subsequently, we cross checked the answers by taking in a few cities from the ‘same state’, Texas. We took Austin, Dallas, Houston, San Antonio, Fort Worth. For this test also, the number of answers provided by the people from the same city was highest. The results we obtained clearly demonstrated that people have a skew towards the same locality. Therefore, to take one more step in this observation, we took top 10 cities (quantitatively) of a country and cross-referenced answers provided by people from the same city and from the rest of the country. For instance, we took Questions from New York and checked answers provided by the people of New York and the rest of the country, i.e. USA excluding New York. This test was conducted for all the previously specified 12 countries. However, in the results, and again the answers provided by people from the same city was high. Evidence for this test is shown in Tables 2 and 3, results for USA and India is presented. For reasons of brevity, the number of test subjects for a few of the cities from USA are shown in Table 4. For this particular observation, although people have the same nationality and clearly the numbers are against the odds, but the results indicate that people prefer answering to a fellow from the same locality more. These statistics are a clear indication that people in SO prefer a person who is similar to them. Through our study of the crowd’s behavior, we realized that the definition of the crowd implies a faceless entity [29], but one has to understand that to the crowd the requester is not faceless (e.g. the one answering a question can see the one asking the question). In this context, there are several studies in sociology investigating the effect of race [30], gender and similarity [31], status [32] on people’s attitude toward others. Moreover, and specially in context of location based helping people, literature has found that people often help others when they belong to the same locality or share similar spatial characteristics [33]. Though the exact reason why people behave this way is unclear, but the evidence we found and prior research in the domain has confirmed this fact [33]. In context of this paper, specifically in open crowd based forums, we expect neutrality, however, the observation presented here clearly negates this notion.

Participation at SO is projected as a game in literature [1, 5]. In this regard, the next observation is concerned with the decreasing activity of good reputation users in the game. In other words, good reputation users showed high activity at SO initially, but they have taken a step back now. Consequently, we wanted to find out why did you stop playing? In our attempt to find the answer to this question, we came across a ‘subjective’ discussion on Meta² (A discussion forum for problems related to SO). However, there is an issue. It has been specified in [23] that the views on Meta are opinionated. Furthermore, sometimes a few posts on Meta show classic symptoms of Emergent Norm Theory [34]. To explain this problem, we quote from [29], ‘..crowds act in terms of group norms but these group norms are a function of the individual leaders.’ Moreover, there is no definite answer yet. As a result, we wanted to find an objective answer to this question. We therefore conducted a survey of SO users having good reputation. In our attempt, a total of 113 emails were sent to ask for the reasons and form different hypothesis. We received a total of 56 emails (some with comments and suggestions), and 34 people participated in our survey. Owing to lack of a medium to contact good reputation users, we could not include more. Moreover, one of the SO community managers advised us to discuss this matter on Meta. However, owing to reasons presented in this paragraph and for issues discussed during our conversation with SO users, for instance ‘anonymity’, ‘negativity against the user’, trolling etc., we chose to investigate the matter ourselves. In our survey, there were multiple high ranking users, current SO moderators, and ex-SO moderators. The question attracted significant discussion, with some users sending us long and detailed comments. During our discussion, we realized that there are several reasons why people are decreasing their activity. Therefore, we grouped the identified explanations into different categories and designed a questionnaire. The questions, designed in an interactive manner, are shown Table 5 (The complete questionnaire in the form of Google Docs can be accessed at the link³). The users were requested to state how strongly they agree or disagree to the possible reason on a scale of 1–5.

The quantitative result for the reasons is shown in Fig. 1. The complete results are also are presented in Table 5. In the same table, the mean and the standard deviation of the reasons is presented. As users’ response can vary, therefore, Correlation Coefficient (PCC) b/w top 33% and last 33% is also shown. It is visible from the Table that the value for PCC is varying across the identified reasons. But, there is one point of interest that has high mean, low std dev, and a positive PCC value: lack of time. It is the top reason to decrease the activity. To emphasize this point, several users pointed out that they have moved in their lives, and no longer have the time to dedicate to SO. Some have moved to better positions in their work (for instance senior management), others have found jobs, some feel they need to dedicate their time someplace else, and so on. Addiction to SO was a phase in life, and they said the phase is now over. Close to lack of time, is the reason, maturity of the users. In this respect, gaining experience with time is an evolutionary human process, therefore, it is natural that users have become picky in their choice of questions. This problem has a strong relationship with yet another reason for decreasing participation, the issue of finding appropriate questions to answer. This particular category blurs the boundary between the human factors and the search algorithms developed by the system. If a search algorithm is able to match the right question for a user, then the issue of maturity and locating appropriate questions to answer will be solved. However, developing a good search algorithm for questions with code and explanation in Natural language, personalized for every user, is non-trivial.

As the survey progressed, we also found that another important reason that caused users to decrease their activity was—There are a lot of people to answer questions. This reason, however, was surprising. During our conversation with SO users, they pointed out, in contrast to the initial days, there are a lot of users now and there is no need for them to provide an answer. One user pointed out that he/she does not want to be just another face in the crowd. In the words of the user, ‘Because there are so many others to answer, there’s a level of bystander effect (why spend the effort when someone else will do it?)’. Another one pointed out that owing to such huge numbers, one has to be the Fastest Gun in the West. In other words, if one is aiming to get a good score, or is trying to get his/her answer accepted, then one has to be the first to answer, no matter whether the quality of answer is good or not. Though, it is understandable that the users do not like this transition at SO, moreover, it also affects the quality of an answer, but, it is expected in human oriented environments that the sheer size and variety of users will not only create diversity, but will also generate competition (sometimes unhealthy). Nevertheless, it is one of the properties of crowd dynamics. Following the footsteps of this issue, we also asked whether New users know how to ask questions or how to accept answers. In this case, the response of users varied. The evidence from Table 5 and Fig. 1 suggests that not all users agree to this reason. 13 out of 34 chose to remain neutral, and 14 agree to this proposition. Moreover, the PCC value for this reason is also negative.

Going by quantity presented in Fig. 1, duplicate questions is also among the top contenders, but the correlation is not strong. Nevertheless, emphasizing on the point of duplicacy, it is an accepted fact at SO that there are questions that are asked several times. Quantitatively speaking, this property can compel some people to decrease their activity as there is nothing new to learn and no new problems to solve. Hence, one can get ‘bored’. In a different context, it has other consequences as well. This consequence will be discussed later in “Plagiarism and taking credit” section.

All the factor discussed in this section are some of the reasons for users to decrease their activity. However, an important point of note discovered during the survey is the factor concerning Quality and duplicacy of questions. There are several posts on Meta that states SO has an increasing number of low quality questions.⁴ Further, users also argue about the increase in duplicate questions. There is also a discussion among SO users that quality is one of the reasons why people do not like participating. To quote from one the those discussions, ‘Yes, because the quality of “new”? questions is going in the toilet. Almost everything I see **** or is a duplicate’. However, our survey neither found the quality of questions nor duplicate questions as a possible reason to reduce participation. Though, for some users it can be a reason to stop playing, but the users we contacted did not agreed to this reason.

One of the wisest traits of human beings is to help other people. However, the question—‘In the real world, when in a difficulty, can we expect help from the person we helped in the first place’, often raises a few doubts. Consequently, taking inspiration from this idea, we wanted to test, can we expect an answer from a person whom we have helped previously. Therefore, the test was conducted in two different ways. First, we wanted to test the idea of quid-pro-quo i.e. I helped you, will you help me? Second, how many users returned to answer in the post of the person they answered to before. To find evidence, we took 1M users and checked to see if we can find any pattern of interest. In this regard, the test concerning Quid-pro-quo did not return any statistically significant result. Out the 1,039,986 users in our dataset, only 134,257 came to settle the status quo for quid pro quo, with an average of 1.0831 answers. Therefore, helping somebody and expecting help in return is certainly not the case in SO. Note, it could also imply that people help others without expecting anything in return. Although, there can be several other reasons for this behavior, the numbers we present clearly negates the idea of Quid-pro-quo. Similarly, for the second test, the statistics did not favored our expectation either. For this case, owing to the volume of data, we took 100,000 users. Out of 100,000 only 6456 returned to answer in the post of a person, they answered to previously. Though, the responder returned to answer an average of 3 questions, but the statistics are not significant enough to make a difference.

The next inspection was motivated to find the role of gender. In this test, we took questions asked by one gender and cross-referenced the answers posted to by another. To this end, we needed to classify SO users as males and females. Therefore, we took Carnegie Mellon University’s name corpus⁵ and classified users based on their ‘display name’. To maintain a balance between males and females, we considered the similar of subjects during analysis. Our dataset had 80,233 males, and 82,351 females. With this experimental setting, the result for the status of the test is shown in Table 6. From the table we see that the number of answers provided by people to the same gender is high. This observation is true both for females and males. Similar to the spatial characteristics of humans, we expected a gender free environment at SO, but the test results show otherwise. After getting these figures we repeated the test 10 more times. We experimented with different number of subjects in each category (we manually and carefully kept the balance between the numbers). However, the result was the same. Users always preferred to answer more to a person of the same gender.

In industrial and academic research, one of worst practices is plagiarism. It is considered unethical. In this regard, we wanted to find out the existence of this malpractice at SO. In other words, the motive is to check whether users copy the answers of other users word-by-word to gain reputation points. To test this hypothesis, we conducted a few experiments. In our experimental settings, we took short answers (having length greater than 45) and sufficiently long answers (with length greater than 2000). We chose these numbers because (1) there is no proper precedent in literature to classify plagiarism. (2) These figures are an indicative of the textual content (and indirectly efforts) in a post. With this setup, we tried to find the existence of plagiarism in SO. During analysis, and to our surprise, indeed we found plagiarized answers. The number of such plagiarized answers with length more than 45 is 1590 and answers having length greater than 2000 is 634. An important point of note here is that even though the answers are online, and sooner or later one is bound to get caught, however, a few SO users ignore this fact and plagiarize answers anyway. To exemplify this, in Fig. 2 we have presented two snapshots of the same answer in different posts. For privacy reasons, most of the data is blacked out. The existence of this behavior confirms the presence of plagiarism.

In the next round of analysis we test to find out whether people use SO as a marketing platform for their own gain. In other words, we check whether users advertise or publicize their own websites, blogs etc. at SO. For instance, a person locates a question, find a solution and post the link of the answer at SO directing the users to his/her website. The motive is simple—To generate traffic at one’s website. In our view, this type of behavior blurs the boundary between ethical and unethical practice. Therefore, we wanted to get into the details of this issue. In this regard, we extracted several statistics that could bring this issue to light. In our analysis, and to filter the noise, we considered users who have provided more than 10 answers. Further, we considered those users whose answers (atleast 50%) contain a link to their website. Lastly, we also filtered the data based on time to answer (TTA) >3 days (i.e. answer coming after three days of posting a question online). There is evidence in literature [26], as well as in the next sub section, that most regular and honest users try to answer questions quickly (median time is 16 min [26]). Therefore, the number (3 days) was chosen to be sure that a user is taking time to find the question, and is posting an answer on his/her website for the mere purpose of promotion. With this setting, the result is shown in Fig. 3. For privacy and reasons of reverse engineering the identity, the number of posts and UserIds are kept hidden. Furthermore, several parts of the data is blacked out. As visible from the figure, there are several users following this type of a practice in SO. For our experiment, we observed a total of 170 users indulging in this behavior. The highest ratio (Answers with URLs to their website/total answers) is 97%, with the same instance (or user) having posts more than 40 (We don’t present the exact number to keep the identity secret). We believe this type of an activity is not professional. SO is the platform where a user has asked a question, and it is the responsibility of the answerer to provide an answer at this venue. Taking the benefit of SO to generate activity at one’s website is not right. However, a few users ignore this fundamental rule and indulge themselves into such practices. For an ideal environment, we expect honesty. However, for real world Q&A forums this is not the case. This finding also questions the behavior of technical users online.

Before beginning the discussion in this subsection, we first show the number of questions answered based on the parameter: time to answer. The result is shown in Table 7. It is visible from the table that a good number of questions are answered within 10 minutes. An important point of note here is a lot of questions are also answered within 59 s (25,543). Though, some answers contain very little text, for instance PostIDs 80299 or 117256. However, for other questions, the observation raises a few doubts. In simple words, long answers coming in that quickly imply instances that are worth investigating. To go into the details, we designed a few tests. In the experiment, we considered answers with body length more than 500, 1000, 1500, 2000 chars resp (long answers). The motive was: If long answers are coming within a matter of seconds, then the answers are written in advance or people are trying to collude with others. The motive here is to gain reputation points. Therefore, we categorized the test in two different ways. First, we took the same person asking and answering a question. Second, we look at instances where two different users ask and answer questions. The results for the two scenarios are shown in Table 8. In the following text, for reasons of brevity, we concentrate only on 0–59 s. It is visible from the Table that there are substantial posts satisfying this criterion (answers coming within 59 s). To be specific, the number is 10,948. In this regard, lets consider the another worst case where length(ans) >2000, i.e. we have a sufficiently long answer. For this case, we present the average score of the question as well as the answer. Recall that score is responsible for increasing reputation. In cases when one who has asked a question provides an answer, the avg score (of question and answer) is 31 and 19 respectively. In other words, when one follows this behavior then one obtains 50 \(\times\) 5 reputation points (31 + 19 for the score of the post (question and answer both), reputation = reputation + score \(\times\) 5). In the other case, i.e. when two different users are participating, then the average score of questions and answers is 44 and 40 respectively, implying an increment of 220 points for the asker and 200 points for the answerer. Such high figures indicate motivation to gain reputation by using unfair means. As discussed in the beginning of this section, getting good score results in good reputation points. Further, it is also known that users of SO seek reputation. The evidence presented here indicates unfair practice to gain fake Internet points.

The evidence in the previous paragraph indicates cases of collusion and use of unfair means to gain reputation points. However, consider the case of the same user. It is possible that a few users want to share their knowledge with others, whereas others are doing this for the mere purpose of gaining reputation points. Unfortunately, to clear this doubt, we would require a manual survey of such users. In this regard, the evidence show classic symptoms of collusion, the chances that any of these users will agree to the point that they wanted to gain reputation by unfair means is unlikely. Moreover, if we think about this observation, we could also ask: Does this behavior of people (specifically, when two different users ask and answer question immediately) effect the overall “perception” and “fairness” at SO? This question remains unanswered in this paper.