Running experiments on Twitter? Don't forget the bug

Just a quick post to point you to an interesting article about tie formation on Twitter – which is also the place where I found this reference, a couple of days ago:

ResearchBlogging.org
Scott A. Golder and Sarita Yardi (2010). Structural Predictors of Tie Formation in Twitter: Transitivity and Mutuality. Proceedings of the Second IEEE International Conference on Social Computing. August 20-22, Minneapolis, MN.

Here I summarize the results:

  • The more followers you have, the more followers you attract (ok, admittedly this doesn’t come as a surprise…);
  • Reciprocity in tie formation doesn’t seem to be due to similarity in interests but, more likely, to some kind of social obligation (well, this is getting more interesting);
  • Self-presentation (pic, bio and location) doesn’t seem to matter, except for location which appears to be negatively correlated to tie formation (now they got my attention…);
  • Transitivity and mutuality predict tie formation if they are taken together, but authors “suggest that a consistent status hierarchy and some level of tie strength drive this effect” (this is definitely worth looking into).

The approach is kind of classic SNA (let’s look at network structure to predict who is connected to whom), but with a nice experimental twist. As you might know, I’m growingly relying on experiments both in vivo and in silico – since our 2008 study on food choice negotiation, and more recently in our ongoing online eating disorder communities. Which explains my curiosity for the web-based one the authors ran from July-August 2009 with users recruited from Twitter’s public timeline. Subjects were invited to rate the profiles of 14 randomly selected users in their 2-degree networks (basically they were asked how much would they be interested in following alter X). A specific interface was designed and, although the number of subjects who actually completed the experiment was small (n=37), the results seem pretty solid.

One thing that is worth mentioning is the way the authors dealt with a software bug that basically cut out the 14th alter from their ego networks. Pretty matter-of-factly they decided that the fact of  leaving the final one out is not expected to introduce any systematic error. There is grounds for agreement on that, from a purely statistical perspective: basically, as far as the alters are selected randomly (and, more importantly, if they are presented to the respondent in succession), it doesn’t matter if there are 13 or 14 of them.

Where things get more complicated is when you ask the question “so, are we sure that we would obtain the same results if we ran this experiment with 13 alters”? And here is where things get a little more complicated in my opinion. Because to be methodologically fair, if one would want to replicate these results, one wouldn’t have to run the same experiment with 13 alters, but rather with 14 alters and the same software bug. The bug becomes a constituent part of the experimental protocol. More, the way I see it, it becomes a summary of all the experimental errors one can encounter. Of course, as pointed out by the authors, the bug can be considered as a systematic error. But it also has a human (as far as the agency of a human programmer is presupposed) and a random dimension to it (if by random event we mean something lacking plan or purpose – i.e. the software was not intended to produce it). Nothing that a correct error analysis wouldn’t fix, for sure. But I like to sit here and ponder the epistemological ramifications of the way computer-based methods (with their inevitable bugs, viruses and scraps of rebellious routines) are impacting the traditional way we make science…

—a