Test–retest reliability of resting-state connectivity network characteristics using fMRI and graph theoretical measures
Highlights
► Test–retest reliability of graph metrics derived from resting-state fMRI. ► Different preprocessing and confound correction methods are examined. ► Moderate overall reliability, but differed between methods. ► Using a broad filter frequency range (0.008–0.15 Hz) yielded best results.
Introduction
The application of graph theory to networks extracted from functional imaging data has proven to be a promising tool to investigate the complex functional structure of the human brain that influences the dynamics underlying cognition and emotion (Bullmore and Sporns, 2009, Eryilmaz et al., 2011, Rubinov and Sporns, 2010). Several studies using different imaging modalities have consistently found the brain to be organized according to small-world properties, making it highly efficient in terms of information exchange while minimizing wiring costs (Achard and Bullmore, 2007, Micheloyannis et al., 2006, van den Heuvel et al., 2008). Further studies focusing on the finer architecture of the human brain network have shown the brain to exhibit an assortative, hierarchical modular structure (Ferrarini et al., 2009, Salvador et al., 2005) (see Materials and methods section for definitions) that can be altered in mental disease (Bassett et al., 2008, Rubinov et al., 2009, Supekar et al., 2008, Wang et al., 2009), is affected by age and sex (Gong et al., 2009, Micheloyannis et al., 2009, Tian et al., 2011) and can undergo rapid plastic changes (Achard and Bullmore, 2007, Polanía et al., 2011). The potential to characterize the entire brain connectome in a way that is both biologically meaningful and related to normal and disordered function across the age span makes these graph theoretical approaches attractive to clinical and basic research. The application of this methodology to resting-state fMRI data is especially interesting since it does not require a cognitive activation task and is therefore, in principle, applicable across the whole age range, at varying levels of awareness and cognitive ability, and even across species.
A variety of different parameters and methods can be used to construct a network graph from functional MRI data. Since a graph is formed by “nodes” (also called vertices or points) connected by “edges” (also called links or lines), the following two steps have to be performed to extract a network graph successfully:
- 1)
Definition of the nodes. In principle each voxel could be a node. In practice, a lower resolution is often preferred. When using a regional template with a resolution lower than the voxel size, additionally an estimator of regional activity is needed.
- 2)
Definition of the links. Two nodes are marked as linked if their respective time series show similarity measures (such as correlation) above a threshold. This requires
- a.
Correction for confounds such as physiological noise and/or global signal.
- b.
Decision about the frequency range.
- c.
Estimation of a measure of connectivity between nodes and definition of a significance threshold for selection of substantially contributing links.
- a.
Steps 1 and 2 are of fundamental importance, since the choice of nodes and links defines the graph and thus determines the network metrics that are then used for neurobiological interpretation (Rubinov and Sporns, 2010). Recently, several studies comparing different strategies for node and edge selection have been published. Fornito et al. (2010) investigated the influence of parcellation scale on different network parameters. They found whole-brain topological parameters to be quite robust over different scales while node specific parameters varied significantly depending on the parcellation scale. Consistent results were reported by Hayasaka and Laurienti (2010), who compared region of interest (ROI)- and voxel-based approaches, arguing for a slight superiority of the latter one. A study of Smith et al. (2011) focused on comparing different methods for the estimation of connectivity between nodes, representing network links. They found methods based on covariance to be quite sensitive to the underlying network, as well as several Bayesian net methods. Moreover, they noted that independently of the connectivity method, the definition of inadequate ROIs (nodes) could seriously compromise network characterization.
From the literature reviewed above, it follows that studies using different node and link selection methods cannot be simply compared quantitatively, especially when considering regional topological metrics based on single node measures.
In the present paper, we investigated two relevant methodological issues: First, we investigated the reliability of topological measures extracted from resting-state fMRI data. This issue is especially important for longitudinal studies or for repeated measurements in the context of a cognitive or drug challenge and lately has received considerable attention (Schwarz and McGonigle, 2011, Telesford et al., 2010).
Second, we studied the influence on these topological measures of different methods to correct for confounds, such as movement or global signal variation, and the choice of different frequency ranges, since these issues have been found relevant in conventional resting-state fMRI analyses (Fox et al., 2009, Murphy et al., 2009, Weissenbacher et al., 2009).
To this end, we systematically explored different estimators of local activity, several confound correction methods and different frequency ranges to construct a network, using the same fMRI resting-state reliability dataset. For each method, we investigated its performance in a two step validating procedure. First, we compared the obtained graph metrics to prominent findings in literature, e.g. whether the networks show small-world properties and are organized in a hierarchical modular fashion. And second, we compared test–retest reliability between methods, using the intra-class coefficient (ICC) as our metric.
Regarding the first step of network extraction outlined above, definition of nodes, we examined differences between estimators of local activity. Using a template with a lower resolution than the original data requires deciding on an estimator of average regional activity. Since most studies and especially those focusing on clinical aspects have used the Anatomic-Automatic-Labeling (AAL) template (Ferrarini et al., 2009, He et al., 2009, Liu et al., 2008, Tzourio-Mazoyer et al., 2002, Wang et al., 2009), we decided on using the AAL node approach as well, providing a certain degree of comparability. Most studies have used the mean time-series as an estimator, but this can be influenced disproportionally by potential outliers. Therefore, using the eigenvariate and median might offer some advantages, especially when using large and inhomogeneous regional templates. For the definition of links (step 2), we examined different methods for correction of confounds (step 2a). The blood-oxygen-level-dependent (BOLD) signal in fMRI is often confounded by fluctuations of non-neural origin (Cordes et al., 2001, Weissenbacher et al., 2009). This can be adjusted for by regressing the time-series of interest against time-series extracted from cerebrospinal fluid (CSF) and white matter (WM).
To address step 2b, we investigated the influence of different frequency ranges on reliability and network metrics. A bandpass filter is often used as an additional procedure to correct for noise. Neural oscillations in resting-state commonly occur at a frequency range between 0.0083 and 0.15 Hz. However, this quite broad frequency range could be obscured by aliased physiological influences like heart beat (0.6 to 1.2 Hz) or respiration (0.1 to 0.5 Hz) (Cordes et al., 2001). Thus, network studies often restrict themselves to smaller frequency bands (Achard et al., 2006, Fornito et al., 2010). We therefore compared the commonly used resting-state frequency range to a more stringent one, often used in network studies.
We did not restudy step 2c (estimation of a measure of connectivity between nodes and matrix thresholding), which is covered in the literature by the study of Smith et al. (Smith et al., 2011). However, as an additional analysis, we investigated the effects of different lengths of time-series on reproducibility of graph metrics. There is a great variety in how many data points are used to estimate the links between network nodes, ranging from 180 to 512 data points per subject. While averaging over long time-series might intuitively provide a better estimator of real connectivity, there are several potential drawbacks of long time-series. First, during resting-state, subjects are told not to focus their attention. However, this gets harder as measuring time increases and different networks become active, which might influence the connectivity measures. Second, while it might be feasible for healthy subjects to lie motionless in the scanner and refrain from external or internal stimuli for a period longer than 5 min, this might not apply to patients, for example suffering from Alzheimer's disease or schizophrenia. Therefore, if network metrics derived from short time-series provide equally reliable results as it was shown for conventional resting-state analysis (Van Dijk et al., 2010), it could also reduce the burden on patients.
Our investigation shows significant impacts of the methodology chosen for steps 1 and 2 on the reliability of topological measures derived from resting-state fMRI. We identified methods associated with reasonably high reliability that we recommend for further use of this approach, especially for longitudinal or interventional studies.
Section snippets
Subjects
A group of 33 healthy right-handed volunteers (mean age: 24.2 years; range: 19 to 29 years; 17 female) was recruited as part of an ongoing study on neurogenetic risk mechanisms for major mood disorders and schizophrenia (Erk et al., 2010, Esslinger et al., 2009) at two collaborating centers in Berlin and Mannheim. 14 participants were scanned in Berlin and 19 in Mannheim. All subjects were scanned twice two weeks apart at the same center. None of the participants reported a lifetime history of
Results
Independently of the estimator used, and across all density levels, all networks showed small-world characteristics indicated by gamma > 1 (gamma = Cnet/Crand) and lambda ≈ 1 (lambda = lnet/lrand) (see Supplementary Fig. 2) and exhibited an assortative, hierarchical and modular structure. The second order metrics studied were higher for lower densities, slowly decreasing as more links were added to the networks. The graphs for all metrics and every method can be found in Fig. 1.
Discussion
One primary goal of this study was to investigate the reliability of graph theoretical measures of human brain functional networks derived from fMRI resting-state data. Overall, we found that reliability was moderate (defined as ranging from 0.41 to 0.59) and highly dependent on the parameters used for the construction of networks and the networks' densities. This is in line with the results reported by Schwarz and McGonigle, who found an overall robust reliability (ICC ~ 0.5), but marked
Acknowledgments
This study was supported by NEWMEDS — Grant Agreement No. 115008. Funding for this study was provided by BMBF (NGFNplus MooDs) and DFG (SFB 636-B7).
Furthermore, we thank Dagmar Gass for her help with data collection and Anna Kopp for proof-reading the manuscript. All other authors reported no biomedical financial interests or potential conflicts of interest regarding this project.
References (62)
- et al.
Effects of model-based physiological noise correction on default mode network anti-correlations and correlations
NeuroImage
(2009) - et al.
fMRI resting state networks define distinct modes of long-distance interactions in the human brain
NeuroImage
(2006) - et al.
Reproducibility of graph metrics of human brain functional networks
NeuroImage
(2009) - et al.
Impact of transient emotions on functional connectivity during subsequent resting state: a wavelet correlation approach
NeuroImage
(2011) - et al.
Test–retest reliability of heart rate variability and respiration rate at rest and during light physical activity in normal subjects
Arch. Med. Res.
(2007) - et al.
Comparison of characteristics between region-and voxel-based network analyses in resting-state fMRI data
NeuroImage
(2010) - et al.
Test–retest study of fMRI signal change evoked by electroacupuncture stimulation
NeuroImage
(2007) - et al.
Using graph theoretical analysis of multi channel EEG to evaluate the neural efficiency hypothesis
Neurosci. Lett.
(2006) - et al.
The impact of global signal regression on resting state correlations: are anti-correlated networks introduced?
NeuroImage
(2009) - et al.
Introducing graph theory to track for neuroplastic alterations in the resting human brain: a transcranial direct current stimulation study
NeuroImage
(2011)
Complex network measures of brain connectivity: uses and interpretations
NeuroImage
Negative edges and soft thresholding in complex network analysis of resting state functional connectivity data
NeuroImage
Network modelling methods for FMRI
NeuroImage
Hemisphere- and gender-related differences in small-world brain networks: a resting-state functional MRI study
NeuroImage
Automated anatomical labeling of activations in SPM using a macroscopic anatomical parcellation of the MNI MRI single-subject brain
NeuroImage
Neural synchrony and the development of cortical networks
Trends Cogn. Sci.
Small-world and scale-free organization of voxel-based resting-state functional connectivity in the human brain
NeuroImage
Effective connectivity of fMRI data using ancestral graph theory: dealing with missing regions
NeuroImage
Correlations and anticorrelations in resting-state functional connectivity MRI: a quantitative comparison of preprocessing strategies
NeuroImage
Efficiency and cost of economical brain functional networks
PLoS Comput. Biol.
A resilient, low-frequency, small-world human brain functional network with highly connected association cortical hubs
J. Neurosci.
Disrupted modularity and local connectivity of brain functional networks in childhood-onset schizophrenia
Front. Syst. Neurosci.
Small-world brain networks
Neuroscientist
Hierarchical organization of human cortical networks in health and schizophrenia
J. Neurosci.
Complex brain networks: graph theoretical analysis of structural and functional systems
Nat. Rev. Neurosci.
Neuronal oscillations in cortical networks
Science
The precision of reliability and validity estimates re-visited: distinguishing between clinical and statistical significance of sample size requirements
J. Clin. Exp. Neuropsychol.
Developing criteria for establishing interrater reliability of specific items: applications to assessment of adaptive behavior
Am. J. Ment. Defic.
Frequencies contributing to functional connectivity in the cerebral cortex in “resting-state” data
AJNR Am. J. Neuroradiol.
Consistent resting-state networks across healthy subjects
Proc. Natl. Acad. Sci. U. S. A.
Brain function in carriers of a genome-wide supported bipolar disorder variant
Arch. Gen. Psychiatr.
Cited by (356)
Striato-cortical functional connectivity changes in mild cognitive impairment with Lewy bodies
2024, Parkinsonism and Related DisordersChildhood trauma is linked to abnormal static-dynamic brain topology in adolescents with major depressive disorder
2023, International Journal of Clinical and Health PsychologyImpact of Amplitude and Phase of fMRI time series for Functional Connectivity Analysis
2023, Magnetic Resonance ImagingNetwork-level dynamics underlying a combined rTMS and psychotherapy treatment for major depressive disorder: An exploratory network analysis
2023, International Journal of Clinical and Health Psychology