Analyzing international organizations: How the concepts we use affect the answers we get

We explore how “international organizations” have been conceptualized and operationalized in the field of International Relations (IR), identify an important gap between the two, and demonstrate how this shapes our understanding of world politics. Traditionally, we show, IR has embraced a broad conception of international organizations (IOs) that appreciates variation in design. However, the literature has largely coalesced around a measurement standard that reflects the characteristics of major postwar IOs. Prevailing measures, therefore, mainly count formal IOs—bodies founded with legally binding agreements—and omit informal IOs, which are founded with non-binding instruments. We argue that this produces a disconnect between theory and empirical evidence used in the field, since scholars frequently make arguments about IOs in general but draw inferences from formal IOs only. After reviewing how this disconnect has emerged, we use an original dataset on state membership in 260 informal IOs to reanalyze a number of important studies, showing heterogeneous effects for subtypes of IOs that conflict with existing theories to varying degrees. These differences imply that formal and informal IOs have different effects and that existing findings in the field are partly artifacts of the specific way IO variables have been operationalized by scholars. Based on this, we offer recommendations for how to improve research practices moving forward.