Skip to main content
SpringerLink
Log in

Is a local sample internationally representative? Reproducibility of four cognitive tests in family dogs across testing sites and breeds

  • Methods Paper
  • Published:
Animal Cognition Aims and scope Submit manuscript

Abstract

A fundamental precept of the scientific method is reproducibility of methods and results, and there is growing concern over the failure to reproduce significant results. Family dogs have become a favoured species in comparative cognition research, but they may be subject to cognitive differences arising from genetic (breeding lines) or cultural differences (e.g. preferred training methods). Such variation is of concern as it affects the validity and generalisability of experimental results. Despite its importance, this problem has not been specifically addressed to date. Therefore, we aimed to test the influence of three factors on reproducibility: testing site (proximal environment), breed and sex (phenotype). The same experimenter tested cognitive performance by more than 200 dogs in four experiments. Additionally, dogs’ performance was tested in an obedience task administered by the owner. Breed of dog and testing site were found to influence the level of performance only mildly, and only in a means-end experiment and the obedience task. Our findings demonstrate that by applying the same test protocols on sufficiently large samples, the reported phenomena in these cognitive tests can be reproduced, but slight differences in performance levels can occur between different samples. Accordingly, we recommend the utilisation of well-described protocols supported by video examples of the whole experimental procedure. Findings should focus on the main outcome variables of the experiments, rather than speculating about the general importance of small or secondary performance outcomes which are more susceptible to random or local noise.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  • Asendorpf JB, Conner M, De Fruyt F, De Houwer J, Denissen JJA, Fiedler K, Fiedler S, Funder DC, Kliegl R, Nosek BA, Perugini M, Roberts BW, Schmitt M, Van Aken MAG, Weber H, Wicherts JM (2013) Recommendations for increasing replicability in psychology. Eur J Pers 27:108–119. doi:10.1002/per.1919

    Article  Google Scholar 

  • Baldini S, Restani L, Baroncelli L, Coltelli M, Franco R, Cenni MC, Maffei L, Berardi N (2013) Enriched early life experiences reduce adult anxiety-like behavior in rats: a role for insulin-like growth factor 1. J Neurosci 33:11715–11723. doi:10.1523/JNEUROSCI.3541-12.2013

    Article  CAS  PubMed  Google Scholar 

  • Brúder I (2010) Development of a test battery for evaluating dog personality and investigating the genetic background of personality traits. Eötvös Loránd University, Budapest

    Google Scholar 

  • Casadevall A, Fang FC (2010) Reproducible science. Infect Immun 78:4972–4975. doi:10.1128/IAI.00908-10

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Crabbe JC, Wahlsten D, Dudek BC (1999) Genetics of mouse behavior: interactions with laboratory environment. Science 284:1670–1672. doi:10.1126/science.284.5420.1670

    Article  CAS  PubMed  Google Scholar 

  • Duranton C, Rödel HG, Bedossa T, Belkhir S (2015) Inverse sex effects on performance of domestic dogs (Canis familiaris) in a repeated problem-solving task. J Comp Psychol 129:84–87. doi:10.1037/a0037825

    Article  PubMed  Google Scholar 

  • Fadel FR, Driscoll P, Pilot M, Wright H, Zulch H, Mills D (2016) Differences in trait impulsivity indicate diversification of dog breeds into working and show lines. Sci Rep 6:22162. doi:10.1038/srep22162

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Fujita K, Morisaki A, Takaoka A (2012) Incidental memory in dogs (Canis familiaris): adaptive behavioral solution at an unexpected memory test. Anim Cogn 15:1055–1063. doi:10.1007/s10071-012-0529-3

    Article  PubMed  Google Scholar 

  • Fukuzawa M, Mills DS, Cooper JJ (2005) The effect of human command phonetic characteristics on auditory cognition in dogs (Canis familiaris). J Comp Psychol 119:117–121

    Article  CAS  PubMed  Google Scholar 

  • Horn L, Marshall-Pescini S, Virányi Z, Range F (2013a) Cross-cultural differences in domestic dogs’ interactions with humans—preliminary results from Ainsworth’s strange situation test. J Vet Behav Clin Appl Res 8:e39. doi:10.1016/j.jveb.2013.04.043

    Article  Google Scholar 

  • Horn L, Range F, Huber L (2013b) Dogs’ attention towards humans depends on their relationship, not only on social familiarity. Anim Cogn 16:435–443. doi:10.1007/s10071-012-0584-9

    Article  PubMed  Google Scholar 

  • Huber L, Racca A, Scaf B, Virányi Z, Range F (2013) Discrimination of familiar human faces in dogs (Canis familiaris). Learn Motiv 44:258–269. doi:10.1016/j.lmot.2013.04.005

    Article  PubMed  PubMed Central  Google Scholar 

  • Huber L, Range F, Virányi Z (2012) Dogs imitate selectively, not necessarily rationally: reply to Kaminski et al. (2011). Anim Behav 83:3–5. doi:10.1016/j.anbehav.2012.03.020

    Article  Google Scholar 

  • Kafkafi N, Golani I, Jaljuli I, Morgan H, Sarig T, Würbel H, Yaacoby S, Benjamini Y (2017) Addressing reproducibility in single-laboratory phenotyping experiments. Nat Methods 14:462–464. doi:10.1038/nmeth.4259

    Article  CAS  PubMed  Google Scholar 

  • Kaminski J, Nitzschner M, Wobber V, Tennie C, Brauer J, Call J, Tomasello M (2011) Do dogs distinguish rational from irrational acts? Anim Behav 81:195–203. doi:10.1016/j.anbehav.2010.10.001

    Article  Google Scholar 

  • Kampis G, Miklosi A, Viranyi Z, Gulyas L (2010) Video deep tagging and data archiving in the comparative mind database. In: Spink AJ, Grieco F, Krips OE, Loijens L, Noldus L, Zimmerman P (eds) Proceedings of the 7th international conference on methods and techniques in behavioral research, Eindhoven. Noldus, Wageningen, NL, pp 185–188

  • Klein RA, Ratliff KA, Vianello M, Adams RB, Bahník Š, Bernstein MJ, Bocian K, Brandt MJ, Brooks B, Brumbaugh CC, Cemalcilar Z, Chandler J, Cheong W, Davis WE, Devos T, Eisner M, Frankowska N, Furrow D, Galliani EM, Hasselman F, Hicks JA, Hovermale JF, Hunt SJ, Huntsinger JR, Ijzerman H, John MS, Joy-Gaba JA, Kappes HB, Krueger LE, Kurtz J, Levitan CA, Mallett RK, Morris WL, Nelson AJ, Nier JA, Packard G, Pilati R, Rutchick AM, Schmidt K, Skorinko JL, Smith R, Steiner TG, Storbeck J, Van Swol LM, Thompson D, Van’T Veer AE, Vaughn LA, Vranka M, Wichman AL, Woodzicka JA, Nosek BA (2014) Investigating variation in replicability: a “many labs” replication project. Soc Psychol 45:142–152. doi:10.1027/1864-9335/a000178

    Article  Google Scholar 

  • Marshall-Pescini S, Frazzi C, Valsecchi P (2016) The effect of training and breed group on problem-solving behaviours in dogs. Anim Cogn 19:571–579. doi:10.1007/s10071-016-0960-y

    Article  PubMed  Google Scholar 

  • Marshall-Pescini S, Valsecchi P, Petak I, Accorsi PA, Previde EP (2008) Does training make you smarter? The effects of training on dogs’ performance (Canis familiaris) in a problem solving task. Behav Process 78:449–454. doi:10.1016/j.beproc.2008.02.022

    Article  Google Scholar 

  • Miklósi Á (2014) Dog behaviour, evolution, and cognition, 2nd edn. Oxford University Press, Oxford

    Book  Google Scholar 

  • Miklósi Á, Topál J (2013) What does it take to become “best friends”? Evolutionary changes in canine social competence. Trends Cogn Sci 17:287–294. doi:10.1016/j.tics.2013.04.005

    Article  PubMed  Google Scholar 

  • Moonesinghe R, Khoury MJ, Janssens ACJW (2007) Most published research findings are false—but a little replication goes a long way. PLoS Med 4:218–221. doi:10.1371/journal.pmed.0040028

    Article  Google Scholar 

  • Müller CA, Mayer C, Dorrenberg S, Huber L, Range F (2011) Female but not male dogs respond to a size constancy violation. Biol Lett 7:689–691. doi:10.1098/rsbl.2011.0287

    Article  PubMed  PubMed Central  Google Scholar 

  • Müller CA, Riemer S, Virányi Z, Huber L, Range F (2014) Dogs learn to solve the support problem based on perceptual cues. Anim Cogn 17:1071–1080. doi:10.1007/s10071-014-0739-y

    Article  PubMed  PubMed Central  Google Scholar 

  • Open Science Collaboration (2015) Estimating the reproducibility of psychological science. Science 349:aac4716. doi:10.1126/science.aac4716

    Article  Google Scholar 

  • Pongrácz P, Bánhegyi P, Miklósi Á (2012) When rank counts—dominant dogs learn better from a human demonstrator in a two-action test. Behaviour 149:111–132. doi:10.1163/156853912X629148

    Article  Google Scholar 

  • Pongrácz P, Gácsi M, Hegedüs D, Péter A, Miklósi A (2013) Test sensitivity is important for detecting variability in pointing comprehension in canines. Anim Cogn 16:721–735. doi:10.1007/s10071-013-0607-1

    Article  PubMed  Google Scholar 

  • Prato-Previde E, Marshall-Pescini S, Valsecchi P (2008) Is your choice my choice? The owners’ effect on pet dogs’ (Canis lupus familiaris) performance in a food choice task. Anim Cogn 11:167–174. doi:10.1007/s10071-007-0102-7

    Article  CAS  PubMed  Google Scholar 

  • Range F, Hentrup M, Virányi Z (2011) Dogs are able to solve a means-end task. Anim Cogn 14:575–583. doi:10.1007/s10071-011-0394-5

    Article  PubMed  PubMed Central  Google Scholar 

  • Richter SH, Garner JP, Würbel H (2009) Environmental standardization: cure or cause of poor reproducibility in animal experiments? Nat Methods 6:257–261. doi:10.1038/NmETH.1312

    Article  CAS  PubMed  Google Scholar 

  • Sidman M (1960) Tactics of scientific research: Evaluating experimental data in psychology, vol 5. Basic Books, New York

    Google Scholar 

  • Topál J, Miklósi Á, Csányi V (1997) Dog–human relationship affects problem solving behavior in the dog. Anthrozoös 10:214–224. doi:10.2752/089279397787000987

    Article  Google Scholar 

  • Tuyttens FAM, de Graaf S, Heerkens JLT, Jacobs L, Nalon E, Ott S, Stadig L, Van Laer E, Ampe B (2014) Observer bias in animal behaviour research: can we believe what we score, if we score what we believe? Anim Behav 90:273–280. doi:10.1016/j.anbehav.2014.02.007

    Article  Google Scholar 

  • van der Staay JF, Arndt SS, Nordquist RE (2010) The standardization–generalization dilemma: a way out. Genes Brain Behav 9:849–855. doi:10.1111/j.1601-183X.2010.00628.x

    Article  PubMed  Google Scholar 

  • Wahlsten D, Bachmanov A, Finn DA, Crabbe JC (2006) Stability of inbred mouse strain differences in behavior and brain size between laboratories and across decades. Proc Natl Acad Sci USA 103:16364–16369. doi:10.1073/pnas.0605342103

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Yang H, Harrington CA, Vartanian K, Coldren CD, Hall R, Churchill GA, Richter SH, Garner JP, Zipser B, Lewejohann L, Sachser N, Schindler B, Chourbaji S, Brandwein C, Gass P, Van Stipdonk N, Wolfer DP, Wu H (2008) Randomization in laboratory procedure is key to obtaining reproducible microarray results. PLoS ONE 3:e3724. doi:10.1371/journal.pone.0003724

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

This research was supported by the European Union and the State of Hungary, co-financed by ESF Research Networking Programme “CompCog”: The Evolution of Social Cognition (www.compcog.org) (06-RNP-020), and the European Social Fund in the framework of TÁMOP 4.2.4. A/1-11-1-2012-0001 ‘National Excellence Program’. Dóra Szabó also received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (Grant Agreement No. 680040), Ádám Miklósi also received support from the Hungarian Academy of Sciences (MTA 01 031), and Friederike Range received financial support from the FWF (Project Number: P24840-B16). We would like to thank Judit Abdai, Camille Hansart, Beáta Korcsok, Krisztina Kovács and Flóra Szánthó for their help. We are also grateful to all owners and their dogs for their participation in the study.

Author information

Authors and Affiliations

  1. Department of Ethology, Eötvös Loránd University, Budapest, Hungary

    Dóra Szabó & Ádám Miklósi

  2. School of Life Sciences, University of Lincoln, Lincoln, Lincolnshire, UK

    Daniel S. Mills

  3. Messerli Research Institute, University of Veterinary Medicine Vienna, Medical University of Vienna and University of Vienna, Vienna, Austria

    Friederike Range & Zsófia Virányi

  4. MTA-ELTE Comparative Ethology Research Group, Budapest, Hungary

    Ádám Miklósi

Authors
  1. Dóra Szabó
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Daniel S. Mills
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Friederike Range
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zsófia Virányi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ádám Miklósi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dóra Szabó.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (CSV 9 kb)

Appendix

Appendix

A priori sensitivity analysis (minimal detectable effect) was carried out with G*Power 3.1.9.2 for a planned sample size of 180 and a power of 0.8 for breed groups and testing sites. The test yielded a sensitivity of 0.052, which meant that our study would be able to detect a medium effect size (η 2 = 0.06) with a power of 0.8 between testing sites and breeds (Table 12).

Detailed scoring of the obedience test

Three types of down commands in a row (only verbal command, only gesture, both). Calling the dog by name was allowed in every case. The next task was a call back from ca. 5 metres, making the dog sit down within a 1-m radius (performance was scored for each of these two subtasks as well as the transition). The last task was a stay command (for 5 s at a distance of 5 metres) after taking a position (sit, lay or stand) of the owner’s choice (each of these two subtasks was scored as well as the transition) (Table 13).

Table 12 Actual sensitivity of the individual subtests regarding testing sites and breed groups
Full size table
Table 13 Inter-observer agreement
Full size table

Task score

4—complete and instant response to the command

3—complete but delayed response to the command, delay to completion not exceeding 5 s.

2—incomplete response to the command; e.g. the dog may not settle in the sit command

1—a nonspecific response to the command; e.g. the dog orients towards the owner

0—no response within 5 s of the command

Transition score

  1. 0-

    no link -link made only after second command

  2. 1-

    link made after pause/delay

  3. 2-

    link made with no delay

Video examples

https://www.youtube.com/watch?v=aaK6NZYh5QQ

https://www.youtube.com/watch?v=Zlhcb36xr8I

https://www.youtube.com/watch?v=rmVd9UDMHcI

https://www.youtube.com/watch?v=KXhTL1NtYcw

https://www.youtube.com/watch?v=T3wWeLth7S8

Video protocols

Means-end test training phase- http://www.youtube.com/watch?v=C5X_QXyJV_E

Means-end test phase- http://www.youtube.com/watch?v=wzpUSW_fRq0

Pointing test: http://www.youtube.com/watch?v=3i35evlPtiw

Problem-solving test: https://www.youtube.com/watch?v=2Kh9PCi5qcY

Memory test: http://www.youtube.com/watch?v=1N316fxBznQ

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Szabó, D., Mills, D.S., Range, F. et al. Is a local sample internationally representative? Reproducibility of four cognitive tests in family dogs across testing sites and breeds. Anim Cogn 20, 1019–1033 (2017). https://doi.org/10.1007/s10071-017-1133-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10071-017-1133-3

Keywords

Search

Navigation