This Article Abstract Full Text (PDF) Services Download to citation manager Citing Articles Citing Articles via HighWire PubMed PubMed Citation

Evaluating Nursing Home Performance Indicators: An Illustration Exploring the Impact of Facilities on ADL Change

Correspondence: Address correspondence to Charles D. Phillips, PhD, School of Rural Public Health, 1266 TAMU, College Station, TX 77843. E-mail: phillipscd{at}srph.tamhsc.edu

	Abstract

TOP Abstract Methods Results Discussion References

 
Purpose: Performance measurement systems for nursing homes assume that facility performance contributes heavily to individual outcomes. This research illustrates how that assumption can be assessed using the change in residents' activities of daily living (ADLs). Design and Methods: The data used in these analyses were all from residents with both an admission and a quarterly assessment in a sample of all admissions to a randomly chosen 10% of Medicare- or Medicaid-certified nursing homes operating during 2002.  Results: Models including both facility and individual variables explained up to 20% of the variation in ADL change after admission. Facility identity in isolation explained between 8% and 14% of the variation in ADL change. Implications: The results suggest that quality indicators based on change in ADLs may be problematic when used in nursing home performance measurement systems. More generally, the results recommend that the level of variation in performance measures attributable to facility identity or performance become a much more central consideration when researchers evaluate quality indicators for use in nursing home performance measurement systems.

Key Words: Nursing homes • Quality indicators • Long-term care • Quality of care

Unfortunately, for a variety of reasons, such indicators are of limited utility in assessing differences among nursing homes. They often have such low prevalence or incidence rates (e.g., burns) that almost all homes share a score of zero. In other instances, data that support these indicators (e.g., medication errors) are difficult to obtain for enough residents to allow one to make reasonable comparative statements about different homes. In other instances, these indicators capture such a narrow slice of the life occurring in nursing homes that one can make little claim that these performance indicators represent well the more general constructs of nursing home quality of care or quality of life.

Given these problems, and the continuing and appropriate desire to assess nursing home performance, researchers often resort to the use of quality indicators that they recognize both are affected by resident characteristics and contain considerable error. Such measures include decline in function in activities of daily living (ADLs), skin condition, weight loss, incontinence, and an array of other measures. These less than ideal indicators are distorted reflections of the unmeasured construct of "pure quality." But these indicators constitute what is available and what seems sensible (Hirdes, Zimmerman, Hallman, & Soucie, 1998). Authors have elsewhere noted a variety of potential problems with these types of measures (Arling, Kane, Lewis, & Mueller, 2005; Mor et al., 2003; Phillips, 2000).

One concern of primary importance, however, is rarely addressed in the discussion of quality indicators. This concern relates to the question, "What amount of variation in an indicator should be attributable to differences in homes' performance in order for the indicator to be useful?" Although it is conceptually an issue of major importance, previous research has paid it little attention. Only with the indicators developed in the Centers for Medicare & Medicaid Services (CMS) project (Morris et al., 2002) on quality indicators did one begin to see such analyses. In that study, the authors considered a quality indicator acceptable if a facility-level model composed of a series of home characteristics resulted in a Pearson's R of at least.45 (i.e., R² =.21).

More recent research has shown that scores on the CMS-developed indicators of quality of life for nursing home residents also prove to be largely beyond a home's control. Research on a summary scale based on these quality-of-life measures found that only approximately 10% of the total variation in scale scores could be attributed to differences among homes (Degenholtz, Kane, Kane, Bershadsky, & Kling, 2006). As potentially unsatisfying as these results may be, they move the field in the proper direction, especially the work by Degenholtz and his colleagues, who directly partitioned variance in outcomes into resident and facility components.

The research presented here investigated the determinants of changes in ADL functioning among nursing home residents. It was an exploration of the degree to which this important outcome, frequently used in measuring the performance of nursing homes, is a function of a home's performance. The results of this research should be of interest to researchers and policy makers who wish to develop measures of a home's performance for use in consumer information systems, survey targeting, or reimbursement in pay-for-performance models.

This effort differs from previous research in that we estimated models that treated each facility as a separate variable. Earlier approaches to this issue focused on specific home characteristics and assumed that those characteristics constituted the facility-level factors that affected resident outcomes. If any unmeasured home characteristics affected outcomes, then the variance attributable to those unmeasured facility characteristics was error in the model.

Our approach did not assume that we could fully identify those specific characteristics of homes that affect resident outcomes. We simply assumed that homes differ in a variety of ways and developed a parameter estimate for each facility that captured, after adjusting for resident characteristics, that home's unique effect on changes in resident ADL status. This approach avoided the potentially unwarranted assumption that we can identify and measure those specific aspects of home operation critical to providing better quality of care. Instead, our approach captured each home's unique effect as a total operational unit on our quality indicator (change in ADL status).

This research also differs from previous research in that we carried out all modeling at the individual or resident levels. The final objective of any performance measurement system is to rank facilities according to the quality of care they provide. To further this goal, researchers often aggregate individual-level data to the facility level (Hirdes et al., 1998). But prior to ranking homes based on their values on some aggregated quality measure, it is important to ask a fundamental question: "How important is facility performance in determining an individual's value on this quality measure?" If an individual's score on the outcome is largely beyond a home's control, then aggregating that outcome to the facility level for use as a facility performance measure may be a questionable enterprise.

	Methods

TOP Abstract Methods Results Discussion References

 
Data
The database used in this research combined data from the CMS Minimum Data Set (MDS) National Archive and the Online Survey, Certification and Reporting (OSCAR) database. The MDS is a multidimensional care screening and assessment tool used to assess every nursing home resident in the country residing in a Medicare- or Medicaid-certified nursing home. Assessments using the MDS or a subset of MDS items occur at admission and quarterly thereafter (Hawes et al., 1995; Morris et al., 1990). MDS admission data for calendar year 2002 and quarterly assessment data for calendar years 2002 and 2003 were the sources for both the dependent variable (change in ADL function) and the individual-level covariates in the multivariate model discussed here.

Information provided by home operators at the time of the home's certification and licensure survey comprise the OSCAR database. In this instance, we used the OSCAR data largely for sampling purposes. The research team took a random sample of 10% of the active nursing home federal provider numbers in the 2002 OSCAR data. The team then matched the Medicare provider number for each of the sampled homes with provider numbers on the individual MDS assessments. All admissions to the sampled homes during 2002 were included in the resident sample.

We used an admission cohort for these analyses because only by using admission data can one develop a set of covariates or an acuity adjustment model free of any facility influence. The home did not cause the resident's status at admission. So, one is free to use any indicator of status at admission that one chooses as a risk adjustor or covariate with no fear of confounding resident status and home performance. In this way, one has a true baseline status that he or she can use to track changes affected by facility performance.

When one uses data on current residents, measures of their status are an interaction of their health status and previous facility performance. A classic example of this problem comes in the analysis of pressure ulcers. Researchers often adjust for whether the resident is bedfast. Residents with limited mobility are at much higher risk for developing a pressure ulcer (Brandeis, Ooi, Hossain, Morris, & Lipsitz, 1994). However, why do residents have limited mobility? Possibly because the home did not engage in an aggressive effort to help residents maintain mobility. So, when one uses current residents and includes having limited mobility in a risk adjustment model for pressure ulcers, one may be adjusting out variation due to poor care as well as resident functional status that is independent of home performance. This is a tangled web of causality that one avoids with the use of an admission cohort, and researchers have recently used this approach elsewhere when evaluating nursing home outcomes (Phillips, Holan, Sherman, Leyk Williams, & Hawes, 2004).

Because our cohort contained all admissions, some proportion of those admissions were readmissions to the same nursing home after an acute-care stay. Twelve percent of admissions came to the home from an acute-care setting and had had a previous stay in the admitting nursing home in the 5 years previous to this admission. Their status at admission was a function of their previous stay in the nursing home and the hospital's performance. To assess the possible impact of readmissions on our results, we first developed our models using all admissions, and we then developed our models using all admissions except those who might have been readmissions (i.e., had had a previous stay in the admitting nursing home and were being admitted from an acute-care facility). This exclusion did not affect our results in any significant way. Using an admission cohort, however, is not without other problems. We discuss these issues later on.

There were an average of 86 admissions per home. The distribution was positively skewed, with a median of 59. Ninety-five percent of the homes had at least 10 admissions during the study period. In our sample from 1,334 facilities, 69% (83,178) of the individuals admitted to a sample nursing home were short stay, whereas 31% (36,584) of the residents were longer stay. Just more than 16,000 facilities were operating in 2002. However, we excluded facilities with fewer than 21 residents from the analysis. This restriction and, more important, problems matching provider numbers between OSCAR and the MDS resulted in sample data from only 1,334 facilities rather than the 1,500+ facilities one would expect from a 10% sample.

Longer stay residents were still present and had a quarterly MDS assessment available. Our analyses focused on the residents with a quarterly assessment. We also performed analyses using imputed ADL scores for residents discharged prior to their quarterly assessment. The results presented here for longer stay residents were very similar to those for shorter stay residents.

Measurement
Dependent Variable
The dependent variable investigated in this study was change in ADL function. We chose this dimension of health status for our illustration because of its importance in understanding an elderly person's abilities, needs, and strengths. As Fillenbaum (2006, p. 7) indicated in the latest edition of the Encyclopedia of Aging, "ADL is central to any assessment of personal independent function. Information on ADL capacity has been used more extensively and for greater variety of purposes than has information from any other type of assessment." Because of this centrality, researchers often include ADL function in systems measuring nursing home performance (Harrington, O'Meara, Simon, & Schnelle, 2003; Zimmerman et al., 1995)

The ADL scale used in this analysis was simply a sum of the scores for seven of these indicators. These ADLs included bed mobility, transfer, locomotion, dressing, eating, toilet use, and personal hygiene. ADL function in the MDS system is measured by indicators of an individual's relative independence (independent, supervision, limited assistance, extensive assistance, total dependence) over the past 7 days in performing eight ADLs. We excluded one indicator, bathing, from our summary scale because function in bathing is as frequently an example of a home's risk management policy as a resident's needs. An individual's score on the scale could range from 0 (independent in all) to 28 (totally dependent in all seven ADLs). Previous research has shown that the MDS ADL scales perform very well when used in a variety of different configurations (Morris, Fries, & Morris, 1999). This research has also shown that ADL data collected during the course of normal nursing home operations are quite similar in quality to those collected in special research data collections (Phillips & Morris, 1997). In these data, the Cronbach's alpha for these seven ADL performance variables was.91, indicating a high level of internal consistency. The dependent variable (change in ADL status) was simply the difference between the ADL scale at admission and at the time of the quarterly admission. Positive values reflect decline, whereas negative values reflect improvement.

Independent Variables
We derived the individual-level covariates included in our analyses from each admission's intake MDS assessment. A number of these covariates were scales. We discussed the ADL scale score in the preceding section; we used its baseline value as a covariate in our models. The individual's cognitive status was measured using the Cognitive Performance Scale. This is a seven-level scale (higher scores indicate more impairment) that has been validated against much longer instruments specifically designed to measure the level of cognitive impairment (Hartmaier et al., 1995; Morris et al., 1994). The covariates also included a modified version of the MDS-Changes in Health, End-stage Disease and Symptoms and Signs (MDS-CHESS), a scale that predicts decline and death among nursing home residents (Hirdes, Fritjers, & Teare, 2003). The original scale itself could not be used because it contains some change items that call for information beyond admission. Instead, the research team entered the components of the MDS-CHESS that it could derive from the admission assessment. Another MDS-based frailty index used as a covariate was the Mortality Risk Index Score (Flacker & Kiely, 2003). Additional covariates included MDS data on the resident's age, gender, and living arrangement prior to entering the nursing home.

Analytic Strategy
Initial steps in the analyses included a review of the distribution and descriptive statistics for each variable. We then developed a series of multivariate models. The dependent variable in these models was change in ADLs in the quarter following admission. We estimated these models at the individual level. The first model included only individual characteristics at admission. The second model included only the identity of the home as a fixed effect. The third model included both individual characteristics and home identity as fixed effects An important aspect in the comparison of these models was a comparison of summary measures of the explanatory power of each model, and in our study the R² statistic served as this measure. We also performed analyses with models in which we entered facility identity as a random effect. The results for these models were very similar to those presented here.

Given our sampling strategy, the standard errors for our variables and the standard errors in the individual-level multivariate models may be reduced slightly due to positive intracluster correlations among nursing home residents in the same home. Given that our main focus was not on individual model parameters and their standard errors, this potential dependence among observations should not affect our general results or conclusions. However, in recognition of this issue, we discuss only those parameters for which p <.01.

	Results

TOP Abstract Methods Results Discussion References

 
Table 1 presents descriptive statistics for our sample. As that table indicates, on average, our residents exhibited slight improvement in their ADL function during their first 3 months in a home. This summary statistic is somewhat misleading, in that residents who declined the most in the first quarter either died or returned to hospital and were not included in our analyses. Two thirds of admissions were women, and the majority of them ranged in age from 75 to 94 years. Eighty-two percent of admissions were non-Hispanic White, with the next largest group of admissions being non-Hispanic Black. The majority of admissions came to the nursing homes from some type of communal living arrangement, with only 26% having lived alone prior to admission. More than half of all admissions (58%) came to the nursing home directly from an acute-care hospital. Close to a majority of admissions had had no episode of long-term care in the past 5 years, but almost 42% had had a previous stay in some nursing home during the 5 years prior to admission to this nursing home.

Because of remaining concern that the first quarter may be atypical, we also estimated our models using those residents who were admitted to the facility and remained in the facility for at least 6 months. For the analysis of those 25,536 residents, the model containing individual-level characteristics only had an R² of.12, the facility-only model had an R² of.10, and the combined model had an R² of.20.

	Discussion

TOP Abstract Methods Results Discussion References

 
This research has a variety of limitations. We looked at only one quality indicator, change in ADL function. Although ADL function is a critical issue in nursing home care and is commonly used, in some form, in reporting systems, our results might have differed if we had looked at some less global quality measure. In addition, the first 6 months after a person is admitted to a nursing home may differ from other periods of residence, but it is a period when one can most clearly analyze variation in outcomes without confusing home and individual effects. Thus, our strategy did not address the cumulative effects of facility performance over a greater number of calendar quarters. Finally, even though we included a wide range of covariates related to resident status, a possibility remains that some form of selection bias might have minimized observed facility effects.

Nonetheless, we began this article by indicating that a good performance indicator is one that is driven largely by the quality of care provided by a nursing home. In essence, variation in individuals' outcomes should be relatively accurately predicted by knowing the identity of the home in which a resident resides. Also, the combination of home and individual characteristics should explain a relatively large proportion of the variation in resident outcomes.

By these criteria, our analyses indicate that changes in global ADL function may be of questionable utility as a performance indicator for nursing homes. Facility identity alone explained, depending on the group, only 8% to 14% of the variation in ADL functional change. Even when we added resident characteristics to facility identity, the total explained variation never exceeded 20%. The factors included in our models did not capture a minimum of 80% of the variation in functional change.

This research focused on resident-level prediction. But, using current resident data at the facility level enabled CMS contract researchers to identify a variety of facility characteristics that gave the investigators a model that explained roughly 26% of the variation in the percentage of chronic care residents who declined in ADL function in a facility (Morris et al., 2002). These researchers considered the ADL measure one of the most valid in their analyses of potential measures. One assumes that they would have had a similar finding if their measure had been one slightly more similar to that in this research, possibly "average ADL change" in a home. In fact, we performed some analyses of the data used in this study with facility-level models, and those models did exhibit somewhat higher levels of explained variation.

So, let us assume that facilities determine between 14% (our estimate) and 26% (CMS study estimate) of the variation in the critical area of resident ADL outcomes. The fundamental question remains: Can one justify the use of a performance measure for nursing homes for which 74% to 86% of the variation in the measure seems to be beyond the control of the home? Measures that show so little dependence on provider action may not be useful elements in performance measurement systems.

Unfortunately, such a judgment at this point is basically an expression of personal taste. Beyond the analysis of ADL change, the broader message of this research is that the field currently lacks a generally agreed-upon standard that would allow one to say that one measure is adequately driven by home performance, whereas another measure is not. This more general issue is of major concern. Current policy discussions emphasize paying for good provider performance and using poor provider performance as a trigger for monitoring or sanctions. It is unclear to us how one does either of those fairly without a clear understanding of how much variation in an indicator can be attributed to provider performance.

Hopefully, this research will stimulate discussion of this issue and the development of some reasonable standard concerning this issue. However, after the field reaches some agreement on how much a home's performance must affect an indicator for that indicator to be useful, investigators must move on and attempt to understand which indicators meet that standard. It may be that facility identity has a much stronger impact of process quality measures than on outcome measures. In addition, another interesting issue becomes whether some homes have a major effect on quality indicators whereas other homes do not. We hope further research addresses these important issues.

	Footnotes

 
The database used in this research was originally developed with funds from the Health Resources and Services Administration, Office of Rural Health Policy under Grant 5U1CRH000033 to the Southwest Rural Health Research Center in the School of Rural Public Health, Texas A&M University System Health Science Center. Michael Sherman's and Min Chen's participation in this research was partially supported by funds from the Biostatistics and Community Health Core of the Center for Environmental and Rural Health, a center at Texas A&M University funded by the National Institute for Environmental Health Sciences under Grant 5 P30 ES009106-08.

¹ Department of Health Policy and Management, School of Rural Public Health, Texas A&M University System Health Science Center, College Station.

² Department of Statistics, Texas A&M University, College Station.

Decision Editor: Linda S. Noelker, PhD

Received for publication September 13, 2006. Accepted for publication April 2, 2007.

	References

TOP Abstract Methods Results Discussion References

 

• Arling, G., Kane, R. L., Lewis, T., & Mueller, C., (2005). Future development of nursing home quality indicators. The Gerontologist, 45, 147-156.[Abstract/Free Full Text]
• Brandeis, G., Ooi, W. L., Hossain, M., Morris, J. N., & Lipsitz, L., (1994). A longitudinal study of risk factors associated with the formation of pressure ulcers in nursing homes. Journal of the American Geriatrics Society, 42, 388-393.[Medline]
• Degenholtz, H. B., Kane, R. A., Kane, R. L., Bershadsky, B., & Kling, K. C., (2006). Predicting nursing home residents' quality of life from external indicators. Health Services Research, 41, 335-356.[Medline]
• Fillenbaum, G., (2006). Activities of daily living. In R. Schulz, L. S. Noelker, K. Rockwood, & R. L. Sprott (Eds.), The encyclopedia of aging: A comprehensive resource in gerontology and geriatrics (4th ed., pp. 6–8). New York: Springer.
• Flacker, J. M., & Kiely, D. K., (2003). Mortality-related factors and 1-year survival in nursing home residents. Journal of the American Geriatrics Society, 51, 213-221.[Medline]
• Harrington, C., O'Meara, J., Simon, L. P., & Schnelle, J. F., (2003). Federal and state nursing facility websites: Just what the consumer needs? American Journal of Medical Quality, 18, (1), 21-37.[Abstract/Free Full Text]
• Hartmaier, S., Sloane, P., Guess, H., Koch, G., Mitchell, C. M., & Phillips, C., (1995). Validation of the Minimum Data Set Cognitive Performance Scale: Agreement with the Mini-Mental Status Examination. Journal of Gerontology: Medical Sciences, 50A, M128-M133.[Abstract]
• Hawes, C., Morris, J., Phillips, C., Mor, V., Fries, B., & Nonemaker, S., (1995). Reliability estimates for the Minimum Data Set for nursing facility resident assessment and care screening (MDS). The Gerontologist, 35, 172-178.[Abstract]
• Hirdes, J. P., Fritjers, D. H., & Teare, G. F., (2003). The MDS-CHESS: A new measure to predict mortality in institutionalized older people. Journal of the American Geriatrics Society, 51, 96-100.[Medline]
• Hirdes, J. P., Zimmerman, D., Hallman, K. G., & Soucie, P. S., (1998). Use of the MDS quality indicators to assess quality of care in institutional settings. Canadian Journal for Quality in Health Care, 14, (2), 5-11.
• Mor, V., Berg, K., Angelelli, J., Gifford, D., Morris, J., & Moore, T., (2003). The quality of quality measurement in U.S. nursing homes. The Gerontologist, 43, 37-46.[Abstract/Free Full Text]
• Morris, J., Fries, B., Mehr, D., Hawes, C., Phillips, C., & Mor, V., et al (1994). MDS Cognitive Performance Scale. Journal of Gerontology: Medical Sciences, 49A, M174-M182.
• Morris, J., Hawes, C., Fries, B., Phillips, C., Mor, V., & Katz, S., et al (1990). Designing the national Resident Assessment Instrument for nursing homes. The Gerontologist, 30, 293-307.[Abstract]
• Morris, J. N., Fries, B. E., & Morris, S. A., (1999). Scaling ADLs within the MDS. Journal of Gerontology: Medical Sciences, 54A, M174-M182.
• Morris, J. N., Moore, T., Jones, R., Mor, V., Angelelli, J., & Berg, K., et al (2002). Validation of long-term and post-acute care quality indicators. Final draft report, Version 2. Cambridge, MA: Abt Associates.
• Phillips, C., & Morris, J., (1997). The potential use of administrative and clinical data to analyze outcomes for the cognitively impaired: Evaluating the Minimum Data Set for nursing homes. Alzheimer's Disease and Associated Disorders, 11, (Suppl. 6), 162-167.[Medline]
• Phillips, C. D., (2000). Measuring and assuring quality care in nursing homes. In L. S. Noelker & Z. Harel (Eds.), Linking quality of long-term care and quality of life (pp. 162–181). New York: Springer.
• Phillips, C. D., Holan, S., Sherman, M., Leyk Williams, M., & Hawes, C., (2004). Rurality and nursing home quality: Results from a national sample of admissions. American Journal of Public Health, 94, 717-1722.
• Zimmerman, D. R., Karon, S. L., Arling, G., Clark, B. R., Collins, T., & Ross, R., et al (1995). Development and testing of nursing home quality indicators. Health Care Financing Review, 16, (4), 107-127.[Medline]

This article has been cited by other articles:

				C. D. Phillips, M. Chen, and M. Sherman To What Degree Does Provider Performance Affect a Quality Indicator? The Case of Nursing Homes and ADL Change Gerontologist, June 1, 2008; 48(3): 330 - 337. [Abstract] [Full Text] [PDF]

This Article Abstract Full Text (PDF) Services Download to citation manager Citing Articles Citing Articles via HighWire PubMed PubMed Citation