Challenges when identifying migration from geo-located Twitter data

For all the attention it attracts, international migration is a minority phenomenon; 97% of the world’s population lives in their country of birth [1]. At the same time, in many developed countries of the global north a non-trivial share of the population is foreign-born ranging from about 10% in Europe, 15% in the US to 20% in Canada and almost 30% in Australia.¹ In most of these countries immigration will be an important aspect of population development for the foreseeable future. Similarly foreign workers, often with precarious legal status, make up a much larger share of the population in some of the Gulf states where in some cases more than three-quarters of the population is foreign born.

Being by definition a mobile population, and, in most countries, constituting a relatively small share of the population, migrants are a hard demographic to study. Although developed states generally have the capacity to collect statistics on their migrant populations, the definition of who is a migrant vary across countries making comparisons difficult, and even in the best of cases there is a substantial lag between migration flows and the availability of migration statistics which makes it especially hard to study and track the sudden population movements that occur in response to political crises. In the global south, where in fact a large majority of the migration occurs, the situation in terms of data availability, accuracy, and access is worse still.

Given these challenges, approaches that draw on digital trace data have sparked the interest of social scientists, governments, and international organizations alike promising to deliver timely estimates of migration flows measured in a consistent way across countries [2, 3]. Beyond studying population movements, some of these data might also provide new ways to study post-migration processes like assimilation and cultural change [4] or even track online hate speech against immigrants [5]. There is now a substantial body of work on this topic and a number methods have been proposed for detecting migrants and migration flows from digital traces [6‐9]. Righi provides a comprehensive summary of work to date, referencing some 50 works in the area [8].

Given this work and excitement about the potential of this data it is surprising, that there seems little in terms of research that directly evaluates whether or not these methods work in the sense that they actually find migrants. When various data and data processing techniques are evaluated this is, with the exception of one paper that coded a small (\(N= 37\)) subset of Twitter users [10], done on the aggregate—that is by comparing estimates of migrant populations or migration trends obtained using digital trace data to estimates based on official statistics or surveys.

Filling this lacuna we use detailed hand-coding to test whether a method proposed in previous research as well as a new, theoretically optimal, method, actually find migrants. Our results are disconcerting: a detailed hand-coding of entire tweet histories shows that both methods we consider have a high false positive rate, classifying what appear to be highly mobile individuals as migrants. Although our results draw from data centered on one country only we have reason to think that the same issues arise in other contexts as well. In fact Canada, given its comparatively high rate of permanent migration, might be a relatively favourable context. We argue that one reason for this high error rate is the fact that migration is a very small share of global human mobility and that twitter users are very select, and skew towards being a highly mobile population.

Researchers have experimented with a number of ways to leverage digital traces to create new sources of data for migrants with each approach presenting unique opportunities but also significant challenges. Principally one can distinguish data that provide information on populations or “groups” but do not allow individual level analysis and data that provide information on individuals. An example of the former is research that leverages the audience estimates generated through Facebook’s advertising platform as a “digital census” of sorts to create estimates of migrant stocks [11], or even more specifically to track changes in interests in migrant populations [4]. This work uses the fact that Facebook classifies expatriate populations by national origin as one of the many advertising audiences. The exact classification however remains a “black box” since both the algorithm and the underlying data are proprietary. And although impressive in its reach and ability to reproduce, with some adjustment, figures from census data, a key limitation of this data is that it does not allow researchers to obtain individual level information constraining the analysis to national-origin or ethnic groups. For example, research that uses these data to study immigrant acculturation [4] has to fall back on group-level analysis, which stands in tension with the state of the art in the social sciences which try to move away from analysis that take groups as the basic constituents of social life [12].

In order to obtain individual level samples of migrants researchers have used a number data sources each representing it’s own challenges. One approach that builds on individual level data uses geolocation of IP addresses. Tracking users through repeated logins on accounts such as web-based e-mail accounts one can then estimate migration flows [13] [14]. Accurate at the country level and in the case of e-mail data with high frequency the data provide a way to track population movements almost in real time. However, the kind of data that contain IP address information are generally proprietary and, as in the case of e-mail, contain sensitive private information. Academic researchers normally do not have access to these data limiting research and especially transparency and reproducibility. Location data gathered from cell-phone records similarly provides high frequency and accuracy but does not provide much in terms of demographic information on the users and comes with similar access challenges and is generally not available to researchers. Since cell-phone networks are generally national these data are, if available, used to examine within-country mobility patterns [15].

Data from social networks are an attractive alternative when they can be gleaned from public profiles and researchers have used the now defunct Google+ social network [16] and LinkedIn [17] to study migration patterns. The most popular source in that regard is geo-located twitter data. The data is publicly available and samples of tweets can be accessed relatively easily through an API. A variety of location data is embedded in tweets and user profiles [18] some of which, as we discuss below, is more reliable than others. Although the Twitter population is clearly a non-random sample of the larger migrant population and even more so when we restrict ourselves to those users who enable the geo-tagging feature in Twitter [19], the unique features of the data one could collect from streams of tweets may well worth be the trade-offs. For population-level research if we can find reasonable adjustment factors we could still infer migration streams at the aggregate level. Even for research that has the aim to study individual migrants the selection bias may be offset by the unique characteristics this data could afford. For example, if it was possible to identify migrants on Twitter and pinpoint their time-window of migration with reasonable certainty one could then analyze Twitter stream before and after migration as an indicator of social involvement in different spheres or even track the network of followers over time.

There is a wealth of literature regarding online user geolocation—usually on Twitter. The task is to identify a user’s location—usually implicitly defined as the user’s home location. There are two general approaches to a solution—social network based [20] and text-based [21]. While current state of the art on this task has advanced to over 98% prediction capability for large-region prediction [22], the task described is not the same as the migration task: neither accommodates changes in location nor integrates the concept of a permanent change in home location as seen in the migration task. Both tasks take a users’ declaration of location, either through the Twitter location field or through geotagged content as a potential solution. To identify the location of the remaining majority of users who do not share such information, social network solutions work by focusing on a user’s friends and followers who have provided this information, making the assumption that a user’s network will have enough co-located users to provide a reasonable inference as to their location. To our knowledge these approaches have so far only been used to identify a single location per user, rather than an explicit permanent or long term change in home location. But, if these data are to form the base for research on migrants or post-migration processes these approaches have liabilities as they include features that are likely to change with the migration and integration process to identify the study population. For example using social networks may systematically miss those migrants that use Twitter to stay engaged with social circles ‘back home’.

Before proceeding it is worth stepping back and consider the larger process by which geo-located twitter data is generated, observed and how that maps onto the challenge of studying migration.

Surprisingly, scholars who study migration rarely define what distinguishes migration from other forms of human mobility. While some make distinctions between short-term (one year or less) and long-term migration, there is generally agreement that migration involves a change of the usual place of residence—at the same time there is broad agreement that legal status and the precise impetus for movement or even the length of stay can note be used as defining characteristics. For example “The UN Migration Agency (IOM) defines a migrant as any person who is moving or has moved across an international border or within a State away from his/her habitual place of residence, regardless of (1) the person’s legal status; (2) whether the movement is voluntary or involuntary; (3) what the causes for the movement are; or (4) what the length of the stay is.” [23]. Yet clearly not all human mobility is migration and at the very minimum when talking about migration we probably want to exclude types of human mobility that do not involve a change of residence such as vacations, business travel, or other short-term mobility.

This heterogeneity in forms human mobility and among the part that we consider international migration overlaps with processes that generate the geo-located data at that is used to make inferences about migration. The probability of a person tweeting and appending geo-location information is clearly not random but highly structured along a number of dimensions including age, socio-economic status, education but also the legal and citizenship status of an individual. Even if just looking at mobile populations it is likely that those fleeing violence, moving undocumented will rarely, if ever, post geo-located tweets. These populations will be systematically missing. On the other hand for those for whom cross-country mobility is a “lifestyle”, leisure travellers and jet setting business executives may be dis-proportionally represented.

Thus two related issues need to be considered when using twitter data to study international migration. First, these data will almost certainly under-represent (or completely lack) large aspects of migration. Second, even if data gleaned from twitter can to be used to study those migrants that generate geo-located twitter data—the second problem, correctly identifying them remains. That is we need to distinguish between migration and other-non migration forms of human mobility (e.g. vacation, business travel) among mobile Twitter users with a reasonable degree of certainty. As we show this is not a simple task and poses both conceptual and data/information issues.

Migration statistics based on administrative data generally draw on data on residence permits or population registries so vacationers in a hotel or visitors to a conference are not included. Most survey-based research on migration similarly sidesteps the issue. Since in almost all cases survey research samples residents only (whether based on registry data, or other sampling frames, random digit telephone dialing etc...) the residence question is already ‘solved’. Conditional on being surveyed, a respondent who was born outside of the survey is simply considered an immigrant.

The aspiration of providing more real-time migration data creates the challenge of making inferences about residence based on information about a persons location history. Several challenges arise: First, since migration is defined as a change in residence this means each person has to have one (and only one) residence for any period of time. Conceptually (and operationally) this makes it hard to classify people whose location history does not suggest one single residence over a period of time. For example, individuals may split their time relatively evenly between two places/countries or they might live “on the road” moving from country to country over extended periods of time. Second, it requires differentiating between mobility that represents a change in residence from mobility that does not. For any one instance of mobility this is a daunting task as fundamentally it goes to the intention behind the move. Any one instance of mobility could be a vacation (or other short-term trip) or a permanent relocation. Ascertaining the intention behind a move from location data alone is generally not possible. In fact assessing intention to change residence or not is a critical aspect of migration control and one that states invest considerable resources in. Modern states have sophisticated bureaucratic processes and documentary regimes [24], and collect large amounts of information in visa applications and visa interviews.

Looking retrospectively at the location histories of individuals makes this problem tractable at least in cases where we have location histories with a high frequency—that is where we can pinpoint the location of an individual at narrow time-intervals (say daily) over long periods of time and where one location is clearly dominant over longer stretches of time (relative to the observation frequency). The sparser the geolocated information and the less clearly one location dominates the location history, the less clearly we can distinguish between mobility that involves a change in residence and mobility that does not.

Geolocated Twitter data is unfortunately not ideal in that regard. Overall, only a small percentage of tweets have embedded geolocation information and even for Twitter users who enable geo-tagging on their Twitter application (volunteered geographic information), geo-located tweets tend to come in “bursts” with often extended time periods where we can’t locate users. Figure 1 illustrates how there are only very few users who do not have large gaps between consecutive tweets—most have at least one large gap. Indeed, 93% of users in our dataset have gaps in their tweet histories of over 30 days, with 99% having gaps of over 5 days, and 63% having gaps of over 4 months. In addition we suspect that many people use the geo-location feature of Twitter not in the same way all the time but are more likely to post their location when traveling than when in their places of residence.

A final challenge for identifying migration is that migration that involves long-term changes in residence is a very small part of global human mobility. If three percent of the worlds population are migrants a much smaller share of the world’s population will migrate in any given year. Current estimates are that in any five year period about 0.5% migrate [25]. In contrast a substantial share of the world’s population will cross international borders as tourists or business travellers. The United Nations World Tourist Organization estimates that there were 1.4 billion international tourists in 2018² almost 20% of the worlds population. The ratio will likely be worse still in the global north where tighter border control means migration is relatively rare and international travel and tourism is much more prevalent than in the global south.

To directly investigate the prevalence and ease of identification of migrants on Twitter, we use human annotators to evaluate results generated by two migration detection algorithms operating at the level of the individual user. We filter our dataset to maximize our chances of finding migrants in a programmatic manner, and then evaluate the potential migration events in a rigorous high-agreement annotation procedure.

To keep the coding manageable and consistent we focus on one case—Canada. In many ways Canada presents a best-case scenario for using twitter data to find migrants. Since it is a country in the global north and due to its selection of highly skilled migrants likely has a relatively larger share of migrants who use twitter. Given that among OECD countries Canada has, relative to its population, one of the largest inflows of permanent migration, the ratios are unlikely to be more favourable in other countries. Still the odds are daunting. A generous accounting would give us just under 850,000 persons who came to Canada on a reasonably long-term basis in 2016. This includes just under 300,000 permanent residents and a roughly equal number of temporary foreign workers (286,000) and counts everyone who holds a foreign student permit (266,000) even though not all students will migrate in one year.³ This number is a small share of the overall number of individuals crossing in and out of Canada in one year. Comprehensive statistics are hard to come by but excluding any trips by immigrants there were about 83 million trips to Canada in 2016.⁴ So in the most generous accounting, migration represents about 1 percent of this mobility.

As this analysis has shown using a variety of coding approaches only a small share of moves classified as changes in residence appear to actually be migration as commonly understood but rather other types of mobility. So, if not migrants, what populations do these approaches capture? Given the relatively small size of the sample we coded in greater detail, our findings here are clearly exploratory. Nevertheless beyond simply mis-classifying vacations and business trips as migrations, some patterns appear in the data.

Given the public nature of Twitter (or at least the unprotected Twitter accounts we were able to code) and the fact that we filter the data for those who have at least one geolocated tweet per month, which in general implies a high frequency of tweeting, it is not surprising that the population we capture tends extrovert. For some a combination of frequent travel and an public facing persona appears to be part lifestyle and part professional freelancers or consultants in creative/cultural industries. Some travel mis-classified as migration was indeed migrants but the mobility picked up was not migration but return trips to visit friends and family.

We also found several examples of people who do not have to appear a single residence at least during the period of our study. One set of examples were people living on the road in the literal sense for example working for musicians on tours. Other examples resembled the ideal-type of the “transnational global citizen” seemingly living in two (or more) countries at the same time and seamlessly moving across borders. Future work could break down these categories similar to the typology identified by Liao et al. [28]. Figure 3 provides an illustration of several such accounts.

The existing work using geo-located Twitter data to study migration has focused on estimating migrant flows between countries for example using geo-located Twitter data as a signal to infer estimates of migration flows using demographic and statistical adjustment models [6]. For this type of work, having a fairly imprecise classification might be acceptable as long as the processed data does provide some signal. That said, if the rate of false positives really is in the range we find in our analysis (between 85 and 95%) then the signal would be quite noisy and a lot of a study’s outcome rides on the method chosen to make inferences.⁶

On the one hand, for studying migrants or post-migration processes, our study suggests that these approaches are likely not useful unless the accurate detection of migrant Twitter users can be improved dramatically. Doing so will prove challenging. If using geotagging approaches, one might require more regular geotagged content—but this will dramatically reduce sample size (by up to 98% according to our study). Imposing longer residence periods will create similar problems—yielding potentially very few accounts classified as potential migrants in the first place.

Further, even if classification issues can be solved, an issue of biased sampling remains. Geotagging based techniques don’t allow us to see those who are too marginalized and vulnerable to leave digital traces. Perhaps more concerning is that they don’t provide ways of estimating the prevalence of those omitted populations.

On the other hand, even if the utility of Twitter data for studying international migration is limited, it might prove useful for studying the types of mobility we saw more frequently in the data and which are, in fact, a much larger part of global mobility than migration, but that received scant attention from social science researchers so far [29]. Studying the life-worlds of these mobile populations could be an important counter-weight in the very migration-focused study of global human mobility and, more generally, provide insights into types of social life that transcend the nation-state [30]. The data that can be gleaned from Twitter accounts might be a good start for such an endeavor especially when combined with other social science approaches such as surveys or targeted in-depth interviews.

This raises a final point, which is that one notable opportunity for future work is the automated disentangling of migrants and otherwise-mobile individuals. Certainly those who migrate and those who are hyper-mobile travel with different intentions—which may well be encoded in the timing, content, and other metadata of tweets. One potentially productive formulation of this exercise would be as a graphical model in which tweets are observations, the latent variable being the type of individual (migrant vs. mobile individual). Such a method, built on top of the methods considered here, could well contribute to both lines of research on global human mobility.