Development and Validation Study of an Instrument to Measure Work Environment of Nurses in Nursing Homes
Article information
Abstract
Purpose
This study was done to develop an instrument to measure the work environment in nursing homes and to evaluate its psychometric properties.
Methods
Instrument development was conducted in three phases: development of the initial questionnaire, content validity and pilot-testing, and construct validity and reliability testing. Participants were 184 nurses from 62 nursing homes in South Korea. Exploratory factor analysis, confirmatory factor analysis, and examination of internal consistency were used to verify the psychometric properties of the instrument.
Results
The final model derived from the factor analysis yielded 27 items across five domains: nurses’ participation in nursing home affairs; well-defined scope of practice; nurse managers’ ability, leadership, and support of nurses; staff and resource adequacy; and communication and coordination. The fit indices in the confirmatory factor analysis were acceptable. Total Cronbach’s ⍺ was .93 (sub-domains ranged .77~.88).
Conclusion
The results indicate that the selected items are suitable for measuring the work environment of nursing homes in South Korea. This instrument may be useful in nursing home-related research and may be used to improve the health outcomes of older adults in nursing homes.
INTRODUCTION
A work environment that supports professional nursing practice is critical for quality of care, residents’ safety, optimal staff performance, and nurse retention. Several studies have shown that a positive work environment for nurses helps improve nurses’ quality of care, patient safety, and satisfaction [1-3], as well as nurse retention [4,5]. However, most previous studies were conducted in acute care settings and few studies have evaluated the work environment in nursing homes. The characteristics of residents in nursing home are unique and the organizational characteristics are distinguished from those of acute care settings. Most residents are frail older people who have chronic health problems and frequently have multiple co-morbidities [6]. Many residents are unable to perform daily activities independently because of cognitive and/or physical impairments [7]. Therefore, nursing home residents require more complicated and sophisticated physical care than do those in acute care settings. They also have complex and interrelated biopsychosocial problems, and they require multifaceted assessment and care [8].
The importance of nursing home care has been emphasized as the number of aged populations with chronic diseases has increased. Consequently, more studies have examined the impacts of work environment on resident and nurse outcomes in nursing homes. For example, Choi et al. [9] found that a supportive work environment is positively related to registered nurses’ job satisfaction in nursing homes. Tourangeau, Cranley, Laschinger, and Pachis [4] showed that work group cohesion and personal accomplishment, which are parts of work environment characteristics, are inversely associated with nurses’ turnover intention. In other studies, the nursing home work environment was inversely associated with the percentage of pressure ulcer [10] and deficiency citations [10,11]. The Swiss nursing home research team has shown that better work environment is positively correlated with care workers’ mental health, while it is negatively correlated with their intention to leave. They also showed that better quality of care was associated with higher teamwork, safety climate, and adequate staffing [12,13].
Nursing work in nursing homes is centered on nursing care whereas nursing work in acute hospital settings focus more on medical treatment. In addition, most of the work in nursing homes is highly autonomous and is accomplished through delegation and cooperation with other nursing staff [6]. As registered nurses in nursing homes have a directive role and responsibilities in their nursing work, registered nurses are required to have greater participation in decision-making to deliver quality of care, better communication and delegation skills between registered nurses and other staff, and good leadership as nurse managers [8]. These characteristics are very different from the work environment of hospital settings where nursing work is more individual and focuses on direct care for residents.
However, most previous studies conducted in nursing homes used the Practice Environment Scale of the Nursing Work Index [14], which was originally developed to measure the work environment in hospital settings [9,10,12], or they employed a combination of several instruments to measure the attributes of nursing work environments [4,11]. Only one developed instrument measures nursing home work environment [15]. That instrument mainly focused on the psychosocial attributes of nursing work environment (i.e., leadership, collaboration, or conflict management); however, it did not include other crucial attributes of a positive nursing work environment such as nurses’ participation in facility affairs, high standards of nursing care, and sufficient nurse staffing and resources [16]. Owing to the complexities of the healthcare system, a multidimensional assessment should be employed to accurately measure the work environment of nursing homes.
As such, the contributors to nursing work environment in nursing homes could differ from that of other healthcare settings, and the existing instrument may not be suitable to measure the unique work environment in nursing homes. Furthermore, to our knowledge, no instrument exists to measure the work environment of nursing homes in Korea. The lack of a usable instrument is a barrier to measuring and improving the work environment of Korean nursing homes. Given the importance of work environment and its potential impact on nursing care quality, a valid and reliable instrument to assess the work environment of nursing homes is desirable. The purpose of this study was thus to develop such an instrument and to evaluate its psychometric properties.
METHODS
1. Study Design
A cross-sectional study was conducted to develop and determine the psychometric properties of the work environment instrument among registered nurses with direct care responsibilities in nursing homes in South Korea. The development and evaluation processes were performed in three phases (Figure 1): Phase 1, development of the initial questionnaire; Phase 2, content validation and pilot-testing; and Phase 3, validity and reliability testing.
Processes of instrument development and psychometric evaluation.
2. Process of Instrument Development and Psychometric Evaluation
1) Phase 1: Development of the initial questionnaire
In the first step, we conducted a literature review on the attributes of nursing home work environment and measurement methods using PubMed, Medline, EMBASE, CINAHL, KoreaMed, Korean Medical Database, and Korean Studies Information Service System; key words included “nursing home” OR “long-term care” AND “work environment” OR “practice environment”. The search was conducted from October 1, 2015 to February 22, 2016; was not limited to English-language papers; and no cut-off date was applied. Identified articles (N=45) were screened by title and then by abstract to exclude irrelevant articles, such as studies that used an instrument for hospital settings without modifying it to measure nursing home work environment. Subsequently, we selected four studies and asked permissions from the original authors to analyze the contents of existing instruments [4,15,17,18]. Overall, 160 items were extracted from the selected studies and six items from a qualitative interview with nurses who worked in nursing homes were added to develop the questionnaire.
The 25 items of the Practice Environment Scale of the Nursing Work Index were also included in the analysis of the new instrument because it is still the most commonly used tool to assess nursing practice environment in various healthcare settings [1-3,9,10,12]. Permission was obtained to use the scale from the original developer, Dr. Lake [14], and authors who translated them into Korean [19]. We removed items that did not directly relate to nursing home work environment or those that overlapped in content and/or meaning. All items were refined for clarity until all three of the current investigators agreed. Assessments and modification were repeated to avoid semantic duplication and to improve clarity of expression for each item.
2) Phase 2: content validity and pilot-testing
The initial version of the questionnaire with 60 items was sent to eight experts to evaluate its content validity. The eight experts included one scale development expert, two content experts, one elder care expert, one nursing home administrator, and three practice nurses in nursing home. All experts assessed whether each item was relevant and appropriate using a 4-point Likert scale (1=extremely irrelevant, 2=irrelevant, 3=relevant, and 4=extremely relevant). The content validity index of the individual items was calculated by taking the percentage of respondents that scored either 3 or 4, and the items with a content validity index of .78 or greater were considered adequate for content validity [20].
To evaluate the clarity and comprehensibility, a pilot test survey using 38 items was conducted with three nurses who provided direct resident care in nursing homes. Then, a focus group interview was conducted with two of the survey respondents to detect items and words that were not understood by the participants as intended by the research team.
3) Phase 3: construct validity and reliability testing
(1) Design, setting, and participants.
In August 2015, there were 2,854 nursing homes nationwide; of these, 1,588 were in Seoul, Gyeonggi, and six metropolitan cities. After excluding 1,122 nursing homes with no nurses, 466 facilities remained. Of these, 107 were facilities with more than 100 beds. To examine the validity and reliability of the newly developed instrument, all 107 facilities were requested to participate in this study. Sixty-four nursing homes agreed to participate and distributed questionnaires to 224 nurses. Finally, questionnaires were collected from 219 nurses from 62 nursing homes. Data were collected from April 4 to July 1, 2016.
After reviewing the collected questionnaires, 35 respondents were excluded because they had one or more items which they did not respond to. The final sample included 184 after excluding incomplete questionnaires. Although there are varying guiding rules about the required sample size in factor analysis (e.g., the sample size should be 100 or greater, the case to variable ratio should be at least 10:1, or the variable to expected factors ratio should be at least 3:1), research findings suggest that the accuracy of the factor analysis is dependent on all features of the gathered data such as the magnitudes of communalities or factor loading, rather than focusing on the sample size alone [21]. Therefore, the sample size of this study was appropriate for the factor analysis.
(2) Data analysis
Statistical analyses were performed using IBM SPSS statistics 23.0 (IBM, Armonk, NY, USA) and STATA version 13 (StataCorp LP., College Station, TX, USA). To test the construct validity of the instrument, an exploratory factor analysis (EFA) and a confirmatory factor analysis (CFA) were conducted. Before the EFA, the Kaiser-Meyer-Olkin (KMO) test and the Bartlett’s sphericity test were used to evaluate whether the data was suitable for factor analysis. The number of factors was explored based on a Kaiser’s criteria (eigenvalue >1 rule), scree plots, and Velicer’s Minimum Average Partial (MAP) test. Principal components analysis (PCA) was conducted with varimax rotation. The factors with a communality of more than .40 and a factor loading of more than .30 were used as consideration criteria. The CFA was performed to verify the goodness-of-fit of the model suggested by the EFA, and the quality of model adjustments was made through the following fit indices: x2 (p>.05), the ratio of minimum discrepancy to the degrees of freedom (CMIN/df; <2 as a good fit), the comparative fit index (CFI; ≥.90 as a good fit), the Tucker-Lewis Index (TLI; .90~.94 as an adequate fit or ≥.95 as an excellent fit), and the Root Mean Square Error of Approximation (RMSEA; ≤.05 as a good fit or .05~.08 as an acceptable fit)[22]. Cronbach’s ⍺ coefficient was used to evaluate the internal consistency: ≥.90 indicates excellent, .80~.89 indicates good, and .70~.79 indicates acceptable internal consistency [23]. Item-total correlation coefficients r >.30 are regarded as being acceptable [23].
3. Ethical Considerations
Ethical approval was secured from the Institutional Review Board of a university (approval number: Y-2015-0024-8). All participants were informed about the study aims and method and provided written consent. The work described in this article was conducted in accordance with the Code of Ethics of the World Medical Association (Declaration of Helsinki).
RESULTS
1. Participants' General Characteristics
Nurses' mean age was 47.2±9.10 years and their mean work experience period was 14.84±7.61 years. Nearly all (99.5%) were women, and 42.4% had obtained at least a bachelor's degree. About half (45.7%) the nurses worked shifts (Table 1).
Participants' General Characteristics (N=184)
2. Instrument Development
Owing to the literature review and qualitative interviews, we developed the initial questionnaire, which comprised 60 items assessing five domains: (1) nurses’ participation in nursing home affairs (10 items); (2) well-defined scope of practice (17 items); (3) nurse managers’ ability, leadership, and support of nurses (7 items); (4) staffing and resource adequacy (11 items); and (5) communication and coordination (15 items). Eight experts evaluated the content validity of the initial questionnaire. Thirty-seven of the 60 items received a content validity index score above .78 and were thus retained. Although one item-“nurses working together talk to each other with an open mind”-scored less than .78, we decided to keep the item with revision because the item was necessary to assess the effectiveness of nurse communication. Based on the pilot survey test and focus group interviews, two items were excluded because of ambiguous meaning in most Korean nursing homes: “quality assurance program” and the absence of “appropriate legal protection system”. Through this process, the preliminary version of the 36-item nursing home work environment instrument was confirmed.
3. Psychometric Evaluation
1) Construct validity
(1) Exploratory factor analysis
The factorability of the 36 items was examined. The KMO measure of sampling adequacy was .89-above the recommended value of .50 [24]-and the result of the Bartlett’s sphericity test was significant (x2=3,247.05, p<.001), indicating that the data were appropriate to perform the EFA.
The PCA using orthogonal varimax rotation revealed the presence of eight factors, with eigenvalues exceeding the Kaiser’s criterion of 1. However, 2 of the 8 factors included only one or two items per factor, and the result of the scree test was also less accurate to determine the number of factors. As the Kaiser criterion tends to result in the over-extraction of factors [25], Velicer’s MAP test was performed to determine the optimal number of factors to retain. From the MAP criterion and the scree test, we determined that a 5-factor solution was more appropriate for this analysis. A varimax rotation was then performed again, and 8 items with low communalities (less than .40) were eliminated from the analysis. The remaining 28 items were entered to the factor analysis. The five factors were named as follows: Factor 1, “nurses’ participation in nursing home affairs”; Factor 2, “well-defined scope of practice”; Factor 3, “nurse managers’ ability, leadership, and support of nurses”; Factor 4, “staffing and resource adequacy”; and Factor 5, “communication and coordination”. Factor loading of the items in the instrument ranged between .39 and .78. The five factors accounted for 59.7% of the total variance (Table 2).
Summary of the Exploratory Factor Analysis of the Nursing Home Work Environment Instrument
(2) Confirmatory factor analysis
Based on the initial EFA model, a CFA was conducted on the same dataset to improve the model fit of the factor structure (Table 2). The results concerning Factors 2 and 3 indicated that the model had acceptable fit to data. We were unable to calculate x2 values for Factor 4, and the x2 values of Factor 1 and Factor 5 were unsatisfactory. However, since x2 is sensitive to large sample sizes, we used several goodness-of-fit indices to evaluate model fit.
The model fit was assessed using CMIN/df, RMSEA, CFI, and TLI. Factor 5 did not reach the cut-off value (CMIN/dF=5.47, RMSEA=.156, CFI=.85, and TLI=.79). This suggested that the results of the factor analysis were invalid and examining the correlations among the items can help improve the fit of the model. We found a high correlation (r=.70, p<.001) between two items-“good work relationship between nurses and other professionals” and “collaboration on resident care with other professionals; e.g., social workers, dieticians, physiotherapists, occupational therapists, speech-language therapists”-within Factor 5. The covariance of these two items was 61.65, indicated a strong relationship between them that overlapped. Therefore, “collaboration on resident care with other professionals” was excluded from the instrument. Consequently, Factor 5 had adequate data fitness (Table 3). The final model derived from factor analysis yielded a 27-item, five-factor model that provided the best fit to the data. The model fit index shows that the model was improved to a good fit except for x2 which is sensitive to sample size: x2=529.50 (p<.001), df=314, CMIN/DF=1.67, RMSEA=.061, CFI=.90, and TLI=.89.
Model fit indices of the Final Model from the Confirmatory Factor Analysis
2) Concurrent validity
Participants were asked to rate the work environment of the nursing home where they worked: “very good or good”, “moderate”, or “bad or very bad”. We compared the single question score to the work environment score measured by the newly developed instrument in this study. The correlation between variables was significant (Cramer’s V=0.196, p=.007). Thus, the concurrent validity of the developed instrument was supported.
3) Reliability
The results of the item analysis to verify the convergence validity of the items and the internal structure indicated that the mean of each item ranged from 2.15 to 3.45 with the standard deviations ranging between 0.67 and 0.98 (Table 4). The corrected item total correlation coefficients were r >.40 (range: r=.44~.75). The Cronbach’s ⍺ for the total 27-item instrument was .93, and Cronbach’s ⍺ of the subscales ranged from .77 to .88. Mean scores for each factor are shown in Table 4.
Item Analysis and Reliability Coefficients of the Nursing Home Work Environment Instrument
DISCUSSION
The work environment in the nursing home affects both older adults and the nurses [12,13]. The purpose of this study was to develop and evaluate an instrument that measures the nursing work environment of nursing homes. Initial questionnaires were developed based on a literature review, qualitative interviews, and items from the Practice Environment Scale of the Nursing Work Index. The validity and reliability of the instrument were analyzed through data collected from 184 nurses working at nursing homes. EFA and CFA were conducted. Overall, the devised scale had 27 items across 5 domains.
The five domains of the newly developed instrument evaluate key attributes of the work environment of nursing homes. Domain 1 revealed the participatory role in nursing home affairs. Nurses have opportunities to be involved in the internal governance of the nursing home and chief nurse officers have authority equal to top-level executives. Domain 2 emphasized the autonomy in providing high-standard nursing care using up-to-date nursing care plans, guidelines, and well-defined roles. Domain 3 focused on the critical abilities of a nurse manager including being a good manager and leader, praising a job well done, and supporting nurses with their problems. Domain 4 described the staffing and resource adequacy of nursing homes. Nurses should be able to spend enough time and resources to provide high-quality care to residents. Lastly, domain 5 was characterized by the relationship with residents, families, and other professionals working together. Increased participation by residents and their families and effective communication and coordination between co-workers improved the quality of care in nursing homes.
The newly developed instrument comprised the same 3 domains that were similar to those of the Practice Environment Scale of the Nursing Work Index [14]: “nurses’ participation in nursing home affairs (domain 1)”, “nurse managers’ ability, leadership, and support of nurses (domain 3)” and “staffing and resource adequacy (domain 4)”. A notable difference between the newly developed instrument and the previous Index was that our instrument reflects the unique work environment of nursing homes, where registered nurses work as healthcare providers who primarily interact with residents. For example, items that can assess nurses’ autonomous nursing activities (e.g., “nursing work has autonomy”, “nurse managers provide clear guidance and advice to nurses”, and “nurse managers respect the opinions of nurses when establishing residents’ care plan”) were included in domain 2, “well-defined scope of practice”. In the Practice Environment Scale of the Nursing Work Index, nurse autonomy was not empirically identified owing to the nature of nursing work and the status of nurses within the hierarchy in hospital settings [14].
In addition, the items regarding the relationship between nurse and doctors in the Practice Environment Scale of the Nursing Work Index were deleted from the new instrument. Instead, items reflecting the construct of communication and coordination with families were included as well as communication with other professions including doctors in nursing homes as domain 5, “Communication and coordination”. The included items within domain 5 assess the role of registered nurses in providing nursing care in nursing homes where various professions work together [26,27]. This is consistent with the results of Temkin-Greener et al. [16], who defined staff cohesion as one of the four domains of the work environment in nursing homes. Temkin-Greener et al. [16] highlighted four psychological factors-leadership, staff cohesion, communication/coordination, and conflict management-to show that executives and nurses are expected to share the values and goals of the facilities and work together to improve the quality of the work environment. Furthermore, as our study was designed to develop an instrument to accurately measure the nursing work environment in nursing homes containing multidimensional constructs, our instrument included not only psychological factors but also system and material factors such as nurses’ participation, clear work arrangements and work instructions, and sufficient nurses and assistive personnel.
Our results revealed the new instrument has adequate construct and concurrent validity and high reliability. The reliability was higher than that of the Practice Environment Scale of the Nursing Work Index when it was developed (Cronbach’s ⍺=.82; subscales ranged .74~.84) [14], but lower than that of the Practice Environment Scale of the Nursing Work Index when it was implemented with Asian nurses (Cronbach’s ⍺=.96; subscales ranged .80~.94) [28]. When compared with the reliability of the Practice Environment Scale of the Nursing Work Index in a study of hospital nurses in Korea, the internal consistency was similar (Cronbach’s ⍺=.93, subscales ranged .80~.84) [19].
The score calculation method of the newly developed instrument was the same as the Practice Environment Scale of the Nursing Work Index [14]; that is, the average of the sub-domains was compared with the median, and the number of sub-domains with an average value higher than the median of the entire domain was counted. Consequently, if the number of sub-domains having an average value higher than the median was 4~5, the work environment is “better”, 2~3 is “mixed” and 0~1 is “poor”. Lake and Friese reported that, when the domain score was less than 2.5, nurses agreed that their working conditions were poor [29]. In the current study, most domains exceeded this cut-off; however, “staffing and resource adequacy” was the lowest (i.e., 2.34 points). This is similar to these results of a study of hospital nurses in Korea (i.e., 2.20 points) [19]. This indicates that the nursing shortage in Korea is dire; in fact, the number of practicing nurses per 1,000 inhabitants in Korea was 6.9 in 2017 and 6.8 in 2016, far lower than the Organization for Economic Cooperation and Development average of 8.9 in 2016 [30].
Nursing work, resident characteristics, and organizational characteristics in nursing homes are unique and differ from those of acute care settings. Hence, this newly developed instrument may help nurses, researchers, and policymakers formally and accurately evaluate the nursing work environment specific to nursing homes. Particularly, this instrument measures multidimensional constructs of the working environment including both psychological and organizational factors. The newly developed instrument can also be used to compare the nursing work environment in Korea to those of other countries after it is cross-culturally adapted. Future research using this valid and reliable instrument would help elucidate the antecedents and consequences of the work environment in nursing homes and provide further information to nursing researchers, executives, and policymakers. Ultimately, the results of future studies would help nurses provide better quality of care through an enhanced work environment.
Although we devised a valid and reliable tool, some limitations should be noted. First, this study was developed for nursing homes in Korea. Therefore, further studies with repeated measurements and modifications may be necessary to test the cultural appropriateness of utilizing the tool in other countries. Second, the relationship between the nursing work environment and resident and/or nurse outcomes was not tested. Future studies are needed to examine the convergent and discriminant validity of the novel instrument. Furthermore, examining its test-retest reliability is recommended to determine measurement stability.
CONCLUSION
A strength of this study is that we constructed initial items based on empirical practice like a literature review and qualitative interviews. The selected items were suitable for measuring the nursing work environment in nursing homes by reflecting the organizational characteristics and the characteristics of residents and nursing activities. Findings from this study showed that the newly developed instrument is a reliable and valid tool to measure the nursing work environment in nursing homes. Furthermore, because the analyzed data were collected from nurses working in 62 nursing homes across this country, the instrument can be applied and generalized across the field. Thus, the availability of this instrument will promote research related to the work environment of nursing homes and can improve the health outcomes of the residing older adults.
Notes
The authors declared no conflict of interest.
Contributed to the conception and design of this study - CE, LK, KJ, CS, and KS; Performed the statistical analysis and drafted the manuscript - CE, MD, LK, KJ, CS, KH, KS, and MA; Critically reviewed the manuscript and supervised the whole study process - CE and MA. All authors read and approved the final manuscript.
Acknowledgements
This work was supported by the National Research Foundation of Korea (NRF) funded by the government of the Republic of Korea (grant number NRF-2014R1A1A3049936; Principal Investigator, E. Cho).
