On Becoming Trauma-Informed: Role of the Adverse Childhood Experiences Survey in Tertiary Child and Adolescent Mental Health Services and the Association with Standard Measures of Impairment and Severity

Tweet

Abdul Rahman, MD, FRCPC; Andrea Perri, MSN;
Avril Deegan, MSW; Jennifer Kuntz, MSW;
David Cawthorpe, MSc, PhD

Perm J 2018;22:17-054 [Full Citation]

https://doi.org/10.7812/TPP/17-054

E-pub: 01/15/2018

ABSTRACT

Context: There is a movement toward trauma-informed, trauma-focused psychiatric treatment.
Objective: To examine Adverse Childhood Experiences (ACE) survey items by sex and by total scores by sex vs clinical measures of impairment to examine the clinical utility of the ACE survey as an index of trauma in a child and adolescent mental health care setting.
Design: Descriptive, polychoric factor analysis and regression analyses were employed to analyze cross-sectional ACE surveys (N = 2833) and registration-linked data using past admissions (N = 10,400) collected from November 2016 to March 2017 related to clinical data (28 independent variables), taking into account multicollinearity.
Results: Distinct ACE items emerged for males, females, and those with self-identified sex and for ACE total scores in regression analysis. In hierarchical regression analysis, the final models consisting of standard clinical measures and demographic and system variables (eg, repeated admissions) were associated with substantial ACE total score variance for females (44%) and males (38%). Inadequate sample size foreclosed on developing a reduced multivariable model for the self-identified sex group.
Conclusion: The ACE scores relate to independent clinical measures and system and demographic variables. There are implications for clinical practice. For example, a child presenting with anxiety and a high ACE score likely requires treatment that is different from a child presenting with anxiety and an ACE score of zero. The ACE survey score is an important index of presenting clinical status that guides patient care planning and intervention in the progress toward a trauma-focused system of care.

INTRODUCTION

The movement toward trauma-informed and trauma-focused psychiatric treatment is an important consideration that requires considerable professional education and adjustment in the system of care. For example, psychiatric treatment generally focuses on specific diseases, disorders, and syndromes, whereas, in actuality, from the perspective of developmental psychopathology, the gateways to diagnostic entities—whether idiopathic or constitutional—are invariably influenced by environmental factors such as early experiences. In the future, the ability to understand and contextualize the report of early adverse or traumatic experiences will fundamentally influence the approach to diagnosis and treatment of all psychiatric entities.¹ This proposition is reminiscent to some extent of the longstanding distinction between the categorical and dimensional approaches to conceptualization. For example, adolescent depression may be efficiently measured as a multidimensional entity.² Such arguments represent a false dichotomy, and more modern algorithmic approaches to diagnosis and treatment take into account the multiaxial nature of mental disorders.^3-7 

In this article, we examine the relationship between Adverse Childhood Experiences (ACE) Study survey scores as an index of trauma and measures of impairment typically used systemwide locally to evaluate the clinical severity of referrals and admissions to the regional Child and Adolescent Addiction, Mental Health and Psychiatry Program (CAAMHPP) in Alberta, Canada. The main purpose of this study was to establish an evidence base, as the first step for becoming trauma-informed and ultimately trauma-focused at a system level vis-à-vis the implementation of the ACE survey.

Even though the link between trauma, human development, and subsequent lifespan adaptation has long been established, a focus on child health policy formation^8-18 to guide the development of trauma-informed and trauma-focused intervention is only beginning to take shape as a standard of care in mental health systems.¹⁹ Although many regional institutions in Canada are moving toward developing policies, guidelines, and education, implementation at the level of patient care lags.

One of the most influential bodies of work advancing the importance of the impact of early adversity in relationship to developmental psychopathology has been the ACE Study,^20-23 especially in respect to overall health status during the lifespan.^24-26 Nevertheless, children’s health and mental health systems’ responses continue to plod, as at the time of the seminal Kirby Report,²⁷ especially within our own catchment. For example, there is evidence of local systemic stigma based on population data analysis, in that children registered in tertiary mental health services receive less emergency and inpatient treatment for their physical disorders after psychiatric diagnosis than do children with no mental disorder, even though they have more physical and biomedical diagnoses at the levels of ambulatory and provincial physician billing.²⁸ 

We report on the development of the evidence base by examining the ACE survey scores in relationship to the established clinical measures of clinical severity, global function, and problem severity collected routinely for children and adolescents referred and accepted for treatment.^29,30 Systemwide implementation of the ACE survey, as a first step, positions CAAMHPP to become an evidence-based, trauma-informed service organization, because ACE survey scores necessarily must relate to clinical outcomes in order to evaluate the effect of trauma-focused interventions in clinical practice. We hypothesized that ACE scores would be positively associated with our standard measures of clinical severity and impairment in addition to demographics (age, sex, family composition) and system variables (repeated admissions). Through this examination, we set the stage for evaluating clinical interventions in relationship to ACE scores and treatment outcomes that may inform our long-term goal of developing effective trauma-focused care.

METHODS

This research was conducted under The University of Calgary Research Ethics Board approval (REB15-1057).

Staff Training and Data Collection

After participation in the Alberta Family Wellness Initiative 2014 Accelerating Innovation Symposium, the CAAMHPP administration decided in December 2014 to implement the ACE survey. After 2 formal staff orientations (N = 300) and dissemination of a plan for electronic survey data collection, voluntary implementation of the ACE survey commenced in November 2015, and mandatory implementation of the ACE survey for each registration began in September 2016.

Staff orientation and training about the ACE Study and survey criteria continued biannually between 2014 and 2017. An online presentation and training manual were developed and disseminated in September 2016, and ACE training was embedded in new staff orientation, ensuring a minimal level of consistency in survey completion and thereby enhancing the potential for collecting reliable and valid data. The ACE total score was entered into the regional information system commencing in November 2015. Subsequently in April 2016, all 10 ACE survey items were embedded in the information system, which also contained the clinical profile, outcomes, and demographic and system-level data from which the data for this study were extracted and analyzed.

Each clinical area collected the data and completed the survey as best fit its practice model. For example, some areas might complete the survey on admission if this fit with their model of engagement or, alternatively, gather the ACE survey information during the course of treatment. Training feedback indicated that the ACE survey items represented information normally collected in the course of assessment and treatment, and the survey presented a way to formalize data collection to inform clinical treatment via the ten nominal categories of early childhood adversity as embedded in the ACE survey. The main message disseminated was that the implementation was not gathering new information; rather the ACE survey represented a novel way of organizing clinical information in relation to patient care. The training process also presented the opportunity to disseminate the education developed by the Alberta Family Wellness Initiative.

Participants

Eligible participants included all postimplementation active-treatment enrollments (not consults or assessments; N = 9329 registrations to the regional tertiary child addiction and adolescent mental health services [CAAMHPP, Alberta Health Services, Calgary Zone, Alberta, Canada]). The registrations consisted of 2464 unique males (mean age = 12.3 years), 3268 unique females (mean age = 14.7 years), and 77 unique others (self-identified sex; mean age = 18.8 years).

Instruments

Measurement of clinical profiles of patient referrals and enrollments in the regional tertiary child addiction and adolescent mental health services are well described; valid and reliable measures collected on referral, admission, and discharge have been employed at CAAMHPP since April 2002, and the admission and discharge Strength/Concern scale were implemented in 2008.³⁰ The following measures constituted the independent variables:

1. Admission and discharge ratings of global function were included, as were admission and discharge ratings of problem severity (Strength/Concern scale), which constitute the measurable treatment plan.^29,30 Improving function and reducing problem severity represent the core objectives of any intervention, and employing long-established published measures of these domains is value added.

2. The Western Canada Waiting List Children’s Mental Health-Priority Criteria Score (WCWL-CMH-PCS) form has been completed regularly for first-time admissions since 2002 regionally.³⁰ The WCWL-CMH-PCS form consists of 18 clinical items capturing the clinician raters’ assessment of urgency and has been described in detail elsewhere.^30,31 This measure of clinical urgency prioritizes clients’ relative position on waiting lists in clinics. The items sum to a total score of 100; however, each item is a relatively independent domain of measurement highly relevant to providing a detailed clinical picture when employed to compare clinically distinct groups or, in this study, the relationship to ACE survey scores.

3. Demographic data (age, sex, family composition) and system variables were collected. For demographics, family composition by increasing clinical risk consisted of 3 groups: a) blended/biological families; b) single parent; and c) blood relatives, foster care, or ward of the government. The system variables were repeated admissions and length of stay. (Other system variables, such as the reason for referral, and diagnosis were not included because these variables did not map efficiently onto the sample construction employed in the final analysis.)

4. The ACE survey total score, the dependent variable in analyses, was the sum of 10 items describing distinct categories of early adversity (http://ACEstudy.org/the-ace-score.html). Many studies testify to the validity of the ACE survey.^21-23,32-34 Two forms of ACE survey total scores (cross-sectional and registration-linked) were employed as detailed in the next 2 sections.

Sample Construction

The present model of care employed in the regional child and adolescent mental health service is episodic, meaning that once enrolled in the system of care, a patient may have a single admission or, depending on need, multiple admissions (eg, emergency, inpatient, ambulatory services, or specialized services). The unit of analysis in the present study was each patient, and patients at CAAMHPP tend to have on average three admissions. One assumption supporting the validity of this approach is that early adversity—measured using the ACE survey—preceded mental disorder and the need for services. Many patients in the served population had been admitted before the implementation of the ACE survey. In construction of the dataset under analysis, baseline (first admission) clinical profiles and demographics were linked to last-admission outcomes data (global function and Strength/Concern ratings). These data were then linked to the ACE survey data (cross-sectional data). Exploratory analyses were conducted to examine the relationship between the models of ACE survey data on the basis of cross-sectional admissions with data linked to all registrations for each patient (registration-linked data).

Data Analysis

Demographic variables were examined regarding potential modification and confounding effects. Sex was found to have a significant relationship with both dependent and independent variables, and hence sexes were analyzed separately. Descriptive statistics included mean, median, count of subsample size, lower 95% confidence interval, and upper 95% confidence interval. Significant differences were identified in the bivariate and multivariable analyses employing factor loadings, regression analysis p values, and, where applicable, comparisons of 95% confidence intervals (z set to 1.96).

Polychoric Factor Analysis

We validated the sample construction and examined the relationship between ACE items and total ACE scores (dependent variables) and important independent variables using polychoric factor analyses (designed for binary data) of the ACE survey items’ factor structure. We also compared the results for the bivariate and multivariable hierarchical models-based ACE survey data linked with clinical profile data on the basis of cross-sectional unique individuals and the registration-linked samples.

Polychoric factor analysis of males (cross-sectional ACE items: n = 831) produced one factor (eigenvalue = 4.98) that accounted for 83% of the variance in the ACE score items for males. For the polychoric factor analysis of registration-linked ACE items for males (n = 3004), one factor (eigenvalue = 4.75) accounted for 77% of the variance in the ACE score items for males.

Polychoric factor analysis of females (cross-sectional ACE items: n = 1387) produced one factor (eigenvalue = 5.23) that accounted for 84% of the variance in the ACE score items for females. The polychoric factor analysis of females’ registration-linked ACE items (n = 5196) had one factor (eigenvalue = 5.51) that accounted for 85% of the variance in the ACE score items for females.

For others (cross-sectional ACE items: n = 39), the sample size was not sufficient to calculate the polychoric factor structure. For registration-linked ACE score item data (ACE items: n = 513), one factor (eigenvalue = 5.62) accounted for 77% of the variance in the ACE score items for others.

Collinearity

In addition to descriptive statistics and bivariate and multivariable hierarchical linear regression analyses, collinearity among the variables was examined because of the high degree of correlation between the continuous and discrete independent variables. Potential collinearity of the independent variables was examined by comparing bivariate and multivariable analyses. Collinearity is indicated in the comparison by sign reversal and inflated standard errors in the b-coefficient in the regression on the dependent variable.³⁵ 

Multivariable hierarchical regression analysis was selected as the optimum method to identify for each sex the most parsimonious model while controlling for collinearity. Multivariable hierarchical regression permits grouping of related variables (as described earlier in the “Instruments” section), which are then added successively to the model. Examination of independent variables contributing significantly to the full models at each stage permitted selection of the reduced parsimonious models. Both full and reduced models were examined for potential collinearity.

There were 400 possible pairwise correlations including the dependent ACE score totals variable, and 361 with the dependent variable omitted. Of these correlations, there were 290 significant correlations (p < 0.05) including the dependent variable and 267 significant correlations omitting the independent variable, indicating a high degree of potential collinearity. An examination of collinearity was undertaken comparing changes in the standard errors and magnitude and sign (positive or negative) of the bivariate analyses results with the standard bivariate regression models for each sex and the full hierarchical regression models.

For each case, in comparison within variables between the bivariate and multivariable models, the standard errors were within 10% and bs were within 20%. Sign changes were observed for both males and females on the WCWL-CMH-PCS items: Danger to self-2, School and/or work-2 and Comorbid medical conditions-2. Sign changes were observed for females on the WCWL-CMH-PCS items: Does the child/adolescent (patient) have problems in the context of the home?-2, Female Prognosis without further intervention-2, Female Children’s global assessment of functioning scale-2, Female Internalize symptoms-2, Female Externalize symptoms/disruptive behavior-2, and Female Comorbid psychiatric conditions-2.

All the independent variables measure a single construct (psychopathy); hence, it may be expected that these would be intercorrelated. Hierarchical linear regression provided a means of grouping the variables in a meaningful way. For example, Strength/Concern and function ratings are composites of the measurable treatment plan (Group 1) and the WCWL-CMH-PCS items (Group 2) groups; additionally, demographic and system variables (Group 3) were grouped for convenience to examine the potential effects of collinearity. Of all the independent variables, when we compared hierarchical regression models (full and reduced) with bivariate regression results, only 2 variables in the full hierarchical model for females provided evidence of collinearity in sign reversal and magnitude change. Externalizing symptoms/disruptive behavior-2 and Comorbid psychiatric conditions-2 shifted from significant in the bivariate model to nonsignificant in the full hierarchical model, but they were not in the final model. On the basis of this examination and approach to modeling,³⁵ multicollinearity was deemed to have minimal effect in respect to the final models. The potential effect of multicollinearity was possible given the high level of intercorrelation among all of the variables. Evidence of the effective multicollinearity comparing bivariate and multivariable models was examined in detail, finding a minimal influence, particularly in the hierarchical regression analysis.

RESULTS

Table 1 compares the proportional distributions of responses for each ACE item collected in cross-section for a single admission (at the time of and during the course of admission) or on the basis of data linkage by each individual patient registration to the clinical profile data with baseline data from the first admission and outcomes data from the last discharge. From Table 1, higher proportions indicate a greater number of the group is closer to the value 1 compared with the value 0. All differences in proportions (sex, ACE item number, proportion) between linked and cross-sectional data were less than 9% within each sex (maximum 8% for item 8 for males and maximum 9% for item 3 for females), notwithstanding that there were a number of within sex (ACE items 1, 2, 4, 6, 9 for males and females) and between sex differences (ACE items 1, 3, 4, 9). Note that there were very few cross-sectional ACE surveys completed for the Other category, but that these linked with 536 unique patient registrations, a ratio of 13.7 compared with 3.6 for males and 3.7 for females. Comparison of the unlinked cross-sectional and registration-linked data groups indicated that the data linkage was reliable for analyzing the relationship of the ACE survey data and the standard clinical measures employed. The clinical relevance of this approach is presented in the Discussion section.

Table 2 shows the polychoric factor structure for the registration-linked ACE item data. All item loadings on factors ranged between 2% and 17% of each other. There were greater differences between the sex-based groups: ACE Item 3 (sexual abuse) loads on the factor for females 45% more than for males, and comparable to others (self-identified sex), sexual abuse loads on the factor for others 40% more than for males. This observation indicates that the factor structure of the cross-sectional and linked data was stable and comparable and that females and others reported sexual abuse more than males did.

The sample of patients with ACE scores on at least 1 admission since implementation consisted of 1098 males of an average age of 11.6 years (standard deviation = 4.6 years), 1686 females of average age 13.9 years (standard deviation = 6.1 years), and 49 others of average age 17.3 years (standard deviation = 7.1 years). The total numbers admitted with outcome measures since implementation was 3727 males, 6133 females, and 560 others.

Sex and family composition were associated (3 groups: biological/stepparent; single parent; and adoptive or foster parent, ward, or blood relative; c² = 48.9, p < 0.0001). Table 3 describes all the dependent and independent variables underpinning the development of a final parsimonious model describing the relationship between 3 groups of independent variables and individual ACE survey score totals. Sexes were considered separately for clarity of comparison on the basis of between-group differences within ACE score and within independent variable distributions.

Table 3 also shows (indicated by a superscript b) the independent variables that were not significantly related to the ACE total score in cross-sectional or linked bivariate analysis. Overall, for the significant bivariate regression relationships, compared with the bivariate regression results of unique patient cross-sectional ACE score totals on the independent variable, the results were similar to the regression on the registration-linked data variables. For males, 24 variables significantly predicted cross-sectional ACE score totals, and 22 variables significantly predicted registration-linked ACE score totals. For females, 28 variables significantly predicted cross-sectional ACE scores, and 26 variables significantly predicted registration-linked ACE score totals. For others, the sample size for cross-sectional examination was too sparse to reliably examine the relationship between ACE scores and the independent variables. For the others with registration-linked ACE score data, 16 variables significantly predicted registration-linked ACE score totals. The results of the bivariate analysis for registration-linked and cross-sectional data indicated that the registration-linked data results are stable and representative compared with cross-sectional data. More importantly, these results indicate that most clinical and system measures were related to the ACE score total, a finding that underpins the centrality of the ACE survey in clinical assessment and potentially treatment.

Model Summary

A hierarchical multivariable regression model was developed to describe the relationship of ACE survey total scores to the independent variables. Table 4 provides a summary of the final model, and the details of the independent variables in relation to the ACE total scores for each sex are shown in Table 5. Model development commenced with 3 groups of registration-linked independent variables resulting in the final reduced model, which differed for males and females: Group 1: Minimum admission and maximum discharge global function and Strength/Concern ratings; Group 2: WCWL-CMH-PCS items; and Group 3: The system variable of number of repeated admissions and the demographic variable of family composition. Overall, the variables remaining in the models accounted for 38% of the ACE total score variance for males and 44% of the variance for females.

Results Summary

1. The proportional distributions of the ACE items (Table 1) and the factor structure of the ACE items (Table 2) were consistent comparing by sex the unique patient ACE scores linked in cross-section with first admissions and last discharges to all patient registration-linked ACE item survey data. For example, there was only one factor and although a number of items were significantly different probably owing to sample size (nonoverlapping 95% confidence intervals), the proportions differed by less than 9%. Differences between males and females were observed in cross-section in Table 1 for ACE items 1, 3, 4, 9; there was only one uniqueness value in Table 2 for ACE item 3 sexual abuse. There were insufficient data to include the cross-sectional data for the Other category, but there were sufficient linked data to examine the ACE survey item factor structure in the registration-linked data. Females and others were distinct from males in the factor loading structure, specifically on Item 3 relating to sexual abuse.
2. Comparison of bivariate cross-sectional and linked dependent and independent variable data indicated that using linked data is acceptable from a statistical perspective.
3. There were insufficient registration-linked data in the Other category to develop a multivariate model. Two final reduced models were presented that identified the most important independent variables related to higher ACE total scores for males and females.
4. The results illustrate that the ACE survey was related to clinical outcomes in a manner making it central to the process of care planning.
5. The relationship of family composition to ACE scores was theoretically meaningful.

DISCUSSION

ACEs are undeniably a developmental risk³⁶ and, by corollary, are central to mental health care, especially for children. Largely because of the ACE Study, many organizations have recognized the importance of trauma-informed and trauma-focused care. Incorporating these features into care systems requires a shift in organizational perspective. This is a shift that translates into the relational space extant between recipients and providers of care. The ability to measure assessment and treatment effectiveness is also a core feature pointing to success or failure.

A palpable relationship exists between ACE scores and independent variables measuring clinical impairment and outcomes. This step of examining ACE scores in relation to clinical profiles and outcomes was important in establishing the evidence base for the clinical utility of the ACE survey. The ability to measure outcomes in relation to ACE score totals as well as ACE items will form the basis for establishing in the future which interventions are effective for particular types of early trauma or adversity.

One advantage is that our system of care has a comprehensive regional information system that integrates valid and reliable measurement of clinical profiles and the effects of interventions.^29,30 By employing this established measurement system, we could test validity and hence the potential utility of the ACE survey. We demonstrated that patients with higher ACE scores also had clinical profiles indicative of greater clinical severity and impairment when analyzed using both bivariate and multivariable methods that took into account the specific nature of the data—cross-sectional patient and registration-linked data.

Implications

The main purpose of this study was to validate the potential clinical use of ACE surveys in the formation and execution of clinical intervention plans that might help not only to focus clinical interventions but also to measure their effects differentially in relation to patients’ particular ACE profiles. In our regional children’s mental health and addiction services, ACE survey scores are now an integral component of assessment and treatment. It will become important to log the types of interventions that are being used with children to gauge the progress in the short-term transition to trauma-informed therapy in view of the long-term (five-year) transition to trauma-focused therapy. Fortunately, we possess an extensive evidence base of clinical information to serve as a background against which interventions and strategies may be compared.^29,30 Implementing mandatory ACE survey completion for every admission provides a mechanism for shifting from a diagnosis-driven, reductionist, medical treatment model to include the trauma-focused dimension of care that is more tuned to the individual’s needs.

Traditionally, psychiatry, like other divisions of medicine, has been driven by diagnostic categories and disease-focused models of care.² On the basis of the present results, trauma-informed and trauma-focused care must be a turnstile upstream of diagnosis and care planning. For example, a child presenting with attention-deficit/hyperactivity disorder or anxiety and a high ACE score will likely require treatment that is different from a child presenting with the same disorder and an ACE score of zero. One practical shift that the process of implementing the ACE survey has led to in our system is the a priori concept among staff that treatment is about “what happened” to a child, rather than “what is wrong” with that child.

Limitations

Clinical profiles based on completion of the WCWL-CMH-PCS form are required only on an individual’s first admission from the community. As a result, and together with incomplete data in some variables across subjects, the sample sizes for variables in the bivariate and multivariable analyses were less than those for the completed number of cross-sectional and registration-linked ACE score profiles. Yet, the final sample sizes were sufficient and representative of the served population.

The results remained relatively stable in relationship to the independent variables, and multicollinearity was evaluated and found to have minimal influence. There are no set or universally accepted ways to manage collinearity.³⁵

The sample construction inflated the sample size. Normally studies simply collect baseline and outcome measures ad hoc. The approach used here is novel in some respects and is therefore unfamiliar, yet it has a basis not only for the clinical model of episodic care but also in the emergence of childhood mental disorder generally as a consequence of adversity.

Using the ACE survey, in and of itself, does not make a system of care trauma informed. There is also a great deal of ongoing education and support required. The ACE survey does provide a linchpin for action that can facilitate trauma-informed and trauma-focused care in a health system.

Last, there were insufficient data to represent the Other category. This group had high ACE scores, and, in the future, it will be important to assess and develop interventions specific to these patients with self-identified sex.

CONCLUSION

The ACE survey score is significantly related to clinical impairment measured on entry to tertiary mental health services. As such, the ACE survey score is an important index of past trauma related to presenting clinical status. Additionally, CAAMHPP has successfully implemented the ACE survey, moving closer to becoming an evidence-based trauma-informed system of care—a necessary step on the path to universal trauma-focused care.

Disclosure Statement

The author(s) have no conflicts of interest to disclose.

Acknowledgments

Kathleen Louden, ELS, of Louden Health Communications provided editorial assistance.

How to Cite this Article

Rahman A, Perri A, Deegan A, Kuntz J, Cawthorpe D. On becoming trauma-informed: Role of the Adverse Childhood Experiences Study in tertiary child and adolescent mental health services and the association with standard measures of impairment and severity. Perm J 2018;22:17-054. DOI: https://doi.org/10.7812/TPP/17-054

References

1.    LeTendre ML, Reed MB. The effect of adverse childhood experience on clinical diagnosis of a substance use disorder: Results of a nationally representative study. Subst Use Misuse 2017 May 12;52(6):689-97. DOI: https://doi.org/10.1080/10826084.2016.1253746.
    2.    Nurcombe B, Seifer R, Scioli A, Tramontana MG, Grapentine WL, Beauchesne HC. Is major depressive disorder in adolescence a distinct diagnostic entity? J Am Acad Child Adolesc Psychiatry 1989 May;28(3):333-42. DOI: https://doi.org/10.1097/00004583-198905000-00006.
    3.    Ferrafiat V, Raffin M, Freri E, et al. A causality algorithm to guide diagnosis and treatment of catatonia due to autoimmune conditions in children and adolescents. Schizophr Res 2017 Jun 25 [forthcoming]. DOI: https://doi.org/10.1016/j.schres.2017.06.036.
    4.    Eisman HD, Fogel J, Lazarovich R, Pustilnik I. Empirical testing of an algorithm for defining somatization in children. J Can Acad Child Adolesc Psychiatry 2007;16(3):124-31.
    5.    Postilnik I, Eisman HD, Price R, Fogel J. An algorithm for defining somatization in children. J Can Acad Child Adolesc Psychiatry 2006 May;15(2):64-74.
    6.    Emerson RW, Adams C, Nishino T, et al. Functional neuroimaging of high-risk 6-month-old infants predicts a diagnosis of autism at 24 months of age. Sci Transl Med 2017 Jun 7;9(393):eaag2882. DOI: https://doi.org/10.1126/scitranslmed.aag2882.
    7.    Kamp-Becker I, Poustka L, Bachmann C, et al. Study protocol of the ASD-Net, the German research consortium for the study of Autism Spectrum Disorder across the lifespan: From a better etiological understanding, through valid diagnosis, to more effective health care. BMC Psychiatry 2017 Jun 2;17(1):206. DOI: https://doi.org/10.1186/s12888-017-1362-7.
    8.    Black JE, Greenough WT, Anderson BJ, Isaacs KR. Environment and the aging brain. Can J Psychol 1987 Jun;41(2):111-30.
    9.    Cicchetti D, Toth SL. A developmental psychopathology perspective on child abuse and neglect. J Am Acad Child Adolesc Psychiatry 1995 May;34(5):541-65. DOI: https://doi.org/10.1097/00004583-199505000-00008.
    10.    Cicchetti D, Richters JE. Examining the conceptual and scientific underpinnings of research in developmental psychopathology. Dev Psychopathol 1997 Spring;9(2):189-91.
    11.    Cicchetti D, Cannon TD. Neurodevelopmental processes in the ontogenesis and epigenesis of psychopathology. Dev Psychopathol 1999 Summer;11(3):375-93. DOI: https://doi.org/10.1017/s0954579499002114.
    12.    Cicchetti D, Toth SL. Social policy implications of research in developmental psychopathology. Dev Psychopathol 2000 Autumn;12(4):551-4.
    13.    Cicchetti D, Blender JA. A multiple-levels-of-analysis perspective on resilience: Implications for the developing brain, neural plasticity, and preventive interventions. Ann N Y Acad Sci 2006 Dec;1094:248-58. DOI: https://doi.org/10.1196/annals.1376.029.
    14.    Cicchetti D. Gene-environment interaction. Dev Psychopathol 2007 Fall;19(4):957-9. DOI: https://doi.org/10.1017/s0954579407000466.
    15.    Greenough WT, Black JE, Wallace CS. Experience and brain development. Child Dev 1987 Jun;58(3):539-59. DOI: https://doi.org/10.2307/1130197.
    16.    Perry BD, Pollard R. Homeostasis, stress, trauma, and adaptation. A neurodevelopmental view of childhood trauma. Child Adolesc Psychiatr Clin N Am 1998 Jan;7(1):33-51.
    17.    Perry BD, Azad I. Posttraumatic stress disorders in children and adolescents. Curr Opin Pediatr 1999 Aug;11(4):310-6. DOI: https://doi.org/10.1097/00008480-199908000-00008.
    18.    Phillips SP, Carver L. Early parental loss and self-rated health of older women and men: A population-based, multi-country study. PLoS One 2015 Apr 1;10(4):e0120762. DOI: https://doi.org/10.1371/journal.pone.0120762. Erratum in: PLoS One 2015 May 21;10(5):e0128168. DOI: https://doi.org/10.1371/journal.pone.0128168.
    19.    Shern DL, Blanch AK, Steverman SM. Toxic stress, behavioral health, and the next major era in public health. Am J Orthopsychiatry 2016 Mar;86(2):109-23. DOI: https://doi.org/10.1037/ort0000120.
    20.    Anda RF, Dong M, Brown DW, et al. The relationship of adverse childhood experiences to a history of premature death of family members. BMC Public Health 2009 Apr 16;9:106. DOI: https://doi.org/10.1186/1471-2458-9-106.
    21.    Felitti VJ. Adverse childhood experiences and adult health. Acad Pediatr 2009 May-Jun;9(3):131-2. DOI: https://doi.org/10.1016/j.acap.2009.03.001.
    22.    Felitti VJ. Long-term medical consequences of incest, rape, and molestation. South Med J 1991 Mar;84(3):328-31. DOI: https://doi.org/10.1097/00007611-199103000-00008.
    23.    Felitti VJ, Anda RF, Nordenberg D, et al. Relationship of childhood abuse and household dysfunction to many of the leading causes of death in adults. The Adverse Childhood Experiences (ACE) Study. Am J Prev Med 1998 May;14(4):245-58. DOI: https://doi.org/10.1016/s0749-3797(98)00017-8.
    24.    Cawthorpe D, Wilkes TC, Guyn L, Li B, Lu M. Association of mental health with health care use and cost: A population study. Can J Psychiatry 2011 Aug;56(8):490-4. DOI: https://doi.org/10.1177/070674371105600807.
    25.    Cawthorpe D. A novel population-based health index for mental disorder. Perm J 2013 Spring;17(2):50-4. DOI: https://doi.org/10.7812/TPP/12-081.
    26.    Chartier G, Cawthorpe D. From ‘Big 4’ to ‘Big 5’: A review and epidemiological study on the relationship between psychiatric disorders and World Health Organization preventable diseases. Curr Opin Psychiatry 2016 Sep;29(5):316-21. DOI: https://doi.org/10.1097/yco.0000000000000270.
    27.    Chodos H, MacLeod JJ. Romanow and Kirby on the public/private divide in healthcare: Demystifying the debate. Healthc Pap 2004;4(4):10-25. DOI: https://doi.org/10.12927/hcpap..16849.
    28.    Ghuttora HK, Cawthorpe D. Treatment of physical disorder in children with mental disorder: A health care utilization study. J Hosp Adm 2014;3(2):24-31. DOI: https://doi.org/10.5430/jha.v3n2p24.
    29.    Novick J, Cawthorpe D, McLuckie A. A measurable treatment plan: Using the Children’s Global Assessment and the Problem Severity scales as outcomes of clinical treatment. J Hosp Adm 2017;6(1):9-15. DOI: https://doi.org/10.5430/jha.v6n1p9.
    30.    Novick J, Cawthorpe D, McLuckie A. The validation of the Western Canada Waiting List Children’s Mental Health-Priority Criteria Score Instrument: 2002-2015 results. J Hosp Adm 2016;5(4):1-8. DOI: https://doi.org/10.5430/jha.v5n4p1.
    31.    Cawthorpe D, Wilkes TC, Rahman A, et al. Priority-setting for children’s mental health: Clinical usefulness and validity of the priority criteria score. J Can Acad Child Adolesc Psychiatry 2007 Feb;16(1):18-26.
    32.    Exley D, Norman A, Hyland M. Adverse childhood experience and asthma onset: A systematic review. Eur Respir Rev 2015 Jun;24(136):299-305. DOI: https://doi.org/10.1183/16000617.00004114.
    33.    Fang L, Chuang DM, Lee Y. Adverse childhood experiences, gender, and HIV risk behaviors: Results from a population-based sample. Prev Med Rep 2016 May 31;4:113-20. DOI: https://doi.org/10.1016/j.pmedr.2016.05.019.
    34.    Felitti VJ. [The relationship of adverse childhood experiences to adult health: Turning gold into lead]. [Article in German]. Z Psychosom Med Psychother 2002;48(4):359-69. DOI: https://doi.org/10.13109/zptm.2002.48.4.359.
    35.    Dormann CF, Elith J, Bacher S, et al. Collinearity: A review of methods to deal with it and a simulation study evaluating their performance. Ecography 2013 Jan;36(1):27-46. DOI: https://doi.org/10.1111/j.1600-0587.2012.07348.x.
    36.    Teicher MH, Samson JA. Annual research review: Enduring neurobiological effects of childhood abuse and neglect. J Child Psychol Psychiatry 2016 Mar;57(3):241-66. DOI: https://doi.org/10.1111/jcpp.12507.