Introduction

Performance measures of a pediatric intensive care unit (PICU) have traditionally been documented by clinical and physical parameters of critically ill children [1–3]. Others have focused on long-term outcomes of children after PICU treatment to evaluate pediatric intensive care, in particular health-related quality of life [4, 5]. Nevertheless, healthcare professionals are not only responsible for healthy outcomes of children. Parents too are part of the integrated care system and they have a right to receive the best care. Looking from a quality of care perspective, parent satisfaction is an important area to evaluate the desirable outcome of healthcare performances [6]. Given the increased awareness of consumer-driven healthcare systems, it is not surprising that patient and family satisfaction is increasingly accepted as an important quality indicator in critical care [7]. It is expected that an increasing number of PICUs already use satisfaction surveys to document their quality of care. Whether these surveys are of an acceptable academic standard is unknown. The reality remains that only two parent satisfaction questionnaires for PICU services are documented in the literature [8, 9].

With respect to the empowerment of parents who experienced a PICU admission of their child, their knowledge is valuable for documenting and improving quality of care. Therefore, a large group of parents were consulted in developing a parent satisfaction instrument together with the Dutch PICU nurses and physicians [10, 11]. Their views on satisfaction-with-care topics were the foundation to create a set of quality indicators measuring the perceived care by the parents. The next step and the aim of this study were to construct and evaluate the reliability and validity of the EMpowerment of PArents in THe Intensive Care (EMPATHIC) questionnaire measuring parent satisfaction.

Methods

This multicenter survey study was designed to assess the psychometric properties of the EMPATHIC questionnaire. All eight PICUs in the Netherlands participated in the study. Data were collected during two cohort periods of 4 months between February and September 2009. The medical ethical review board of the Erasmus Medical Center in Rotterdam approved the study followed by approval of the review boards of the participating medical centers.

Questionnaire construction

The item generation and selection of the questionnaire followed a structured process. The first steps concerned item generation. A literature review was performed to generate items from the existing satisfaction surveys [12]. The following step was a Delphi study to identify the importance of the items among 269 PICU nurses and physicians including the identification of possible new items [10]. This study resulted in 78 individual satisfaction-with-care items. Subsequently, a convenience sample of parents (n = 559) recruited from seven PICUs was asked to rate the importance of these 78 items [11]. Also, in-depth interviews with 41 parents were held to explore their experiences. Qualitative analysis revealed 63 subthemes divided into 6 themes. Following the item generation steps, item selection was based on the priorities of the opinions of parents and healthcare professionals. Of the generated items, 60 were found to be most relevant for measuring parent satisfaction with care. The items were rephrased into statements appealing to parents. Several items (n = 18) were considered important to differentiate between nurses and physicians. For example, the item about emotional support given to parents was rephrased into two statements; the first measuring the nurses’ emotional support and the second measuring the physicians’ emotional support. The draft questionnaire contained a demographic section to obtain characteristics of the child and its family, the 78 statements, four control questions about overall satisfaction-with-care, and free space was made available for the parents to write comments on their experiences. A 6-point scale, from 1 ‘certainly no’ to 6 ‘certainly yes’, was used for the statements. Parents could also tick a box if the statement was not applicable. This draft was introduced to eight nurses and eight physicians of the EMPATHIC study group and to ten parents. The aim was to investigate whether the statements were understandable and clearly formulated. The feedback resulted in some textual adjustments to the statements. The questionnaire was then translated into Arabic and Turkish by registered translators since a considerable number of parents might have difficulty in reading and understanding the Dutch questionnaire.

Statistical analyses

Descriptive statistics were used to calculate mean scores and standard deviations of the statements. Before analyzing the psychometric properties of the instrument, four negatively formulated statements were deviated from the other statements. Two statements were deleted because more than 75% of the parents scored these as not applicable leaving 72 statements for analysis. For each statement principal component analysis for the non-numerical data was performed to determine the scale value of the answer alternative ‘not applicable’. Two-dimensional plots of the individual statements revealed that this value is closely related to the highest score on the 6-point scale. Therefore, it was justified to impute the answer alternative ‘not applicable’ to the highest value (in this case 6), meaning that parents who ticked the answer alternative ‘not applicable’ belong to the highest category of the 6-point scale.

Structural equation modeling is an advanced multivariate analysis technique using various analysis methods to identify, test, and estimate the relationship between study variables. Confirmatory factor analysis (CFA) is tailored to unraveling the empirical structure of the interrelationship of the 72 statements. The final model was based on both theoretical and statistical plausibility. The measures applied in this study were χ2 test of model fit, and the ratio of \( {\frac{{\chi^{2} }}{df}}\, < \,3 \) represents a good model fit. Other tests used for the model fit were: comparative fit index (preferably CFI ≥ 0.95), Tucker–Lewis index (preferably TLI ≥ 0.95), root mean square error of approximation (preferably RMSEA <0.08), and the weighted root mean square residual (preferably WRMR < 0.90) [13].

Reliability in this study refers to the internal consistency of the statements within the instrument. Internal consistency reliability measures were performed on the individual statements of the domains identified. On domain level the Cronbach’s α was assumed to be satisfactory with values between 0.70 and 1.00. In order to get insight into the stability of the findings over time, the means of the domains at two measurement moments were tested for difference using the Levene’s test for equality of variances and t test for equality of means. If no statistical difference occurred, it was considered that the reliability over time was adequate.

Validity refers to the degree in which an instrument measures what it is intended to measure. Content validity and face validity were ensured as described by the questionnaire construction above. Congruent validity was assessed by using the Spearman’s rank correlation for estimating the relationship between the statements on domain level and four overall satisfaction-with-care scales. Furthermore, the non-differential validity referring to variables, which were assumed to have non-differential effects, was calculated by using the effect size of standardized mean difference (Cohen’s d) between the demographic variables and the domains. The effect size is small with a value of 0.20, medium with 0.50, and large with greater than 0.80 [14].

All significance testing was set at P ≤ 0.05 (two-tailed). SPSS (version 15, Chicago, USA) and the statistical modeling program Mplus (version 5, 2007, Los Angeles, USA) were used for analysis.

Participants

The sample size was aimed at 600 parents at least for adequate psychometric analysis and a second cohort of 600 parents to test the stability of the instrument over time. Parents whose child was discharged alive from a PICU were recruited. If a child had one or more PICU admissions in the study period, the parents received only one questionnaire at the first admission. Parents whose child had died were excluded because in the Netherlands these parents receive an invitation to meet with the PICU physician and nurse 6 weeks after their child’s death. During this meeting all aspects of treatment and care are evaluated. Two to three weeks after PICU discharge parents received an invitation letter, including the information about the study, the EMPATHIC questionnaire, a consent form to be signed, and a prepaid reply envelope. One reminder was sent after 3 weeks if no response was received. Ethnicity of the parents was determined by asking the cultural background of the family in the demographic section of the questionnaire. In the analysis ethnicity was categorized into Dutch and non-Dutch.

Results

In the first 4-month data collection period the questionnaire was mailed to 1,055 parents and 667 (63%) parents responded. In the second 4-month period 551/991 (56%) parents responded. Between both cohorts of parents, the questionnaire was completed by almost similar percentages of mothers (61.5 vs. 64.3%), fathers (11.9 vs. 11.4%), both mother and father (24.3 vs. 23.2%), and others (2.3 vs. 1.1%). The characteristics of the children and parents of the two cohorts are presented in Table 1 and they are not significantly different.

Table 1 Characteristics of children and parents
Full size table

During the process of structural equation modeling, the CFA showed that seven statements did not fit into the empirical structure of the domains. These statements where therefore removed from further analysis. Overall, the performance of the model fit of the remaining 65 statements in the five domains was adequate (Table 2). The standardized factor loadings of the statements were above 0.40 except for two statements (Table 3). The statement about the physician’s knowledge of the child’s medical history had a standardized factor loading of 0.39 and the statement about having an assigned physician during PICU stay had a standardized factor loading of 0.36.

Table 2 Performance of the models
Full size table
Table 3 Mean, standard deviation, and standardized factor loadings of satisfaction statements (n = 667)
Full size table

The reliability estimates of the individual statements and domains were sufficient. On the domain level the reliability estimates, Cronbach’s α, were between 0.73 and 0.92 in the first cohort of parents. These estimates were also calculated among the second cohort of parents and showed similar outcomes from 0.73 to 0.93 (Table 4). At different time moments the two cohorts did not significantly differ on the equal variances for differences of the means on domain level and differences of variances (Table 4). Additionally, the standardized mean differences on domain level, Cohen’s d, were calculated between the two cohorts: information −0.02, care and cure −0.09, parental participation −0.12, organization −0.12, and professional attitude −0.11. Overall, the reliability and stability of the EMPATHIC questionnaire over time provided empirical evidence that the reliability of the instrument was adequate.

Table 4 Descriptives, reliability estimates, and testing on domains of two cohorts
Full size table

The EMPATHIC questionnaire showed sufficient correlation with the four statements of overall satisfaction as an indicator of congruent validity (Table 5). The non-differential validity of the instrument was assessed by calculating the effect size of domains and various population variables. The four tested variables of the five domains did not show significantly differences (Table 6). The parents of a child with a surgical procedure admission were more satisfied about the information statements than parents whose child had a medical PICU admission, Cohen’s d 0.29, P < 0.001.

Table 5 Congruent validity of scales assessed using Spearman’s rank correlation
Full size table
Table 6 Non-differential validity, differences between domains and PICU characteristics
Full size table

Discussion

The strength of this study lies in the rigorous development of the satisfaction instrument through a convenience sample of parents and PICU healthcare professionals. Psychometric testing of the EMPATHIC questionnaire revealed that this instrument is reliable and valid for measuring parent satisfaction of pediatric intensive care services. The 65 statements representing five domains provide a comprehensive conceptualization of parent satisfaction. The relevance of this study is not limited to the development of an acceptable and feasible quality performance indicator for PICUs. Healthcare departments are increasingly demanding satisfaction outcomes of individual healthcare services above and beyond general hospital satisfaction measures [15–17]. Besides, healthcare insurance companies increasingly demand patient satisfaction data to fund hospitals [18]. This study provides a valuable contribution relevant to PICU colleagues in other countries and cultures. The increased need for validated instruments to collect data of patient or family experiences is evident. The challenge is to construct and design research instruments for use in study populations with different cultural backgrounds. The EMPATHIC questionnaire showed that there were no significant differences between Dutch parents and parents with a non-Dutch background. However, these results can not be generalized when using the EMPATHIC questionnaire in other countries with other languages. Although its reliability and validity has been proven, thorough translation of the instrument is needed to provide evidence that the meaning of the translated statements is equivalent to the original statements [19]. For this translation process, a framework has been described of 10 consecutive steps, based on a review of 12 guidelines for translation and cultural adaptation, to ensure principles of good practice [20].

Only two validated parent satisfaction instruments have been documented in the literature for PICU settings [8, 9]. The instrument developed in the UK assessed the face and content validity although no reliability testing was performed [8]. Despite the proper testing of the reliability and validity of the 23-item instrument developed in the USA, the authors stated that a limitation of this instrument is its restricted number of items [9]. The authors stated that their instrument was not intended to be ‘all-inclusive’ but that it was rather directed to PICU care only. However, it is debatable whether 23 statements are sufficient to reflect PICU services. In contrast, after its validation process the EMPATHIC questionnaire included 65 statements. In this perspective, two issues are important to address: the number of statements needed to measure the complexity of PICU services and the length of the questionnaire related to the response rates. The latter is addressed in a review of 210 patient satisfaction studies. In 125 studies the number of items used in the survey was reported and ranged from 1 to 361 items [21]. Of these, there was a weak negative correlation between the number of items and the response rate. A similar effect occurred between the Picker Patient Experience (PPE-15) questionnaire embedded in a four-page survey with 31 questions including demographic details and a 12-page survey with 108 questions [22]. The analyses revealed that the length of the questionnaire did not lead to reduced response rates and there was no effect on the quality of data in terms of percentages of items completed. Therefore, the 65-statement EMPATHIC questionnaire seems feasible as it resulted in response rates of 63 and 56% in the two measurement periods. Incidentally, these response rates can be considered acceptable for satisfaction surveys because the mean initial response rate was 72.1% (SD = 19.8) in 124 satisfaction studies [21]. The other issue to be addressed is the question of how many statements are needed to measure parent satisfaction for PICU services. Basically, an instrument should include items on all factors of importance to the trait under study. This refers to the content validity of an instrument. A review of ten satisfaction instruments designed for PICU, neonatal ICU, adult ICU, and pediatric wards documented a range of 15–45 items [12]. The strength of the EMPATHIC questionnaire lies in the item generation and item selection phase where a large number of parents and healthcare professionals were consulted to identify the most important items [10, 11]. The items were converted into statements and theoretically conceptualized in domains related to family-centered care concepts [23]. It is, therefore, recognized that the 65 statements of the EMPATHIC questionnaire reflect the most important issues of the PICU care.

Two limitations of the psychometric testing of the EMPATHIC questionnaire need to be addressed. The first concern is the test–retest reliability. This refers to administering the same instrument to the same respondents at two different moments in time to estimate its stability over time. A high correlation between the two measurement periods refers to a good test–retest reliability. It was decided not to burden the parents with two questionnaires in a short period of time. Therefore, as it was assumed that neither the care and the treatment nor the population of parents would change within 8 months, two cohorts of parents were included to test the stability of the findings over time.

The second limitation is the lack of criterion validity testing of the instrument. This validity refers to association of the results between the tested instrument and another validated instrument measuring the related concept. Although one parent satisfaction instrument for the PICU was tested to be valid [9], its limitations to use it as the gold standard were based on a single center study, the limited items of the instrument, and the validity tested on a small group of 40 parents of which 27 responded. For that reason, congruent validity was chosen to examine part of the validity of the EMPATHIC questionnaire by testing the statements on four generally accepted gold standard questions measuring overall satisfaction.

Despite the recognition that satisfaction outcome measures are increasingly used as quality performance indicators [24, 25], the reality is that there is a paucity of PICU parent satisfaction instruments in the literature. However, it is expected that many PICUs have developed and carry out their own unit-based satisfaction survey. Therefore, and in conclusion, this study provides a reliable and valid parent satisfaction instrument that is feasible to administer to parents. The found empirical structure of the satisfaction-with-care statements can be considered adequate. The proven statistical evidence of the statements is of importance in order to be able to apply the EMPATHIC questionnaire in other PICUs. Nevertheless, including statements of clinical relevance specific to local PICU settings might be considered. Ultimately, using a validated instrument contributes to the empowerment of parents to work collaboratively with nurses and physicians on PICU quality of care.