The Gerontologist 47:838-844 (2007)
© 2007 The Gerontological Society of America
Mode of Administration Effects on Disability Measures in a Sample of Frail Beneficiaries
Edith G. Walsh, PhD1 and
Galina Khatutsky, MS1
Correspondence: Address correspondence to Edith G. Walsh, PhD, RTI International, 1440 Main Street, Suite 310, Waltham, MA 02451-1623. E-mail: ewalsh{at}rti.org
 |
Abstract
|
|---|
Purpose: To compare disability rates resulting from several modes of survey administration in a single sample of frail elders. Design and Methods: Using the same battery of six ADL questions we compared the resulting level of disability across several modes of administration: mail survey with telephone follow-up, in person interview, and evaluation by a registered nurse, further comparing self and proxy responses where both were available. We also created a crosswalk between these measures and clinical evaluations by rehabilitation therapists, allowing another point of comparison. Results: Disability rates varied substantially by mode of survey administration and all survey modes yielded lower rates of disability than those we derived from clinical assessments. Implications: Relying on self-report in evaluating functional status may underestimate disability in clinical evaluations, level of care determinations and service planning. Researchers and policymakers should also take mode of administration effects into account when estimating or comparing disability rates.
Key Words: Health outcomes survey • Core outcomes and comprehensive assessment • Program of All-Inclusive Care for the Elderly (PACE) • Risk adjustment • Frailty factors
Researchers and practitioners use disability rates derived from functional status measures for many purposes (McDowell, 2006). Researchers routinely collect functional status measures through survey and health provider assessments because they (a) provide valuable information on prevalence rates of disability, (b) support estimates of the economic burden of illness, and (c) are used extensively for planning medical, rehabilitation, and support services. Experts also use functional status data for payment, for example, as part of case-mix payment systems in long-term care (Kautter & Pope, 2004; Weissert & Musliner, 1992). Case-mix reimbursement refers to payment rates that reflect the actual mix of patients in a facility based on some measures of acuity. In long-term care, case-mix reimbursement systems rely heavily on functional status measures along with diagnostic information, rehabilitation services, and nursing treatments. In addition, recent advances in risk adjustment methodology have added frailty measures derived from functional status as a component to diagnosis-based data (risk adjustment predicts the future expenditures of an individual based on diagnoses and demographics; Kautter & Pope, 2004; Pope, Adamache, Walsh, & Khandker, 1998). Functional status measures also serve as important control variables in survey and outcomes research.
Previous research has provided ample evidence for the following: Disability rates differ according to the definition of impairment within or across data sets (e.g., Jette, 1994; LaPlante, 1991), self-report and caregiver reports differ (e.g., Todorov & Kirchner, 2000), and people respond differently to in-person surveys compared to telephone or mail surveys (see Dillman & Christian, 2005, for a review). In general, people report better health in person than when asked by telephone, and they report the lowest levels of health in mail surveys. Looking specifically at activities of daily living (ADLs) and instrumental ADLs, Gruenberg, Kaganova, and Corazzini-Gomez (2001) found a higher probability of reporting a condition through a mail survey than in person. In an earlier study, Gruenberg, Tompkins, and Porell (1989) found that clinically determined ADLs and instrumental ADLs differ considerably from those obtained through self-report. McHorney, Kosinksi, and Ware (1994) also found that mail-survey respondents report greater impairment and poorer health compared to telephone-survey respondents. Prior research comparing self- and proxy reports has found that proxies generally rate people as more impaired than individuals rate themselves, although there is less disagreement about physically based observable measures (compared to other quality-of-life measures) and less disagreement about ADLs than instrumental ADLs (e.g., Jones & Feeny, 2005; Ostbye, Tyas, McDowell, & Koval, 1997). Despite these differences, using proxy reports is essential to capturing data about those with substantial impairments or cognitive deficits. In a Health Outcomes Survey–Modified (HOS-M) analysis (Walsh, Nason, Moore, & Khatutsky, 2003), the authors compared the survey results to medical record data from the Program of All-Inclusive Care for the Elderly (PACE) and found differences in agreement by respondent type. For example, compared to medical records, self-respondents reported substantially lower levels of impairment, whereas family proxy reports were very close to medical record information.
However, the literature includes few comparisons of multiple data collection approaches within the same sample. Studies have typically compared responses by two different samples to each mode (e.g., comparing responses from those who answered a mail survey with the responses of mail-nonresponders who were interviewed by telephone). To address this gap, we compared ADL impairments in a sample of 547 frail elders (a population with a high rate of functional impairment) by using a unique data set that contains functional status measures collected via two modes of survey administration: a mail survey with telephone follow-up and face-to-face interviews. We then compared survey data with functional status assessments conducted by registered nurses (RNs) and rehabilitation therapists and, where possible, compared self- and proxy reporting.
This study used data derived from two pilot projects conducted in overlapping PACE sites of PACE. Before describing the study methods and our findings, we provide brief background information about these activities.
|
PACE
|
|---|
PACE is a health care delivery model integrating long-term care with the Medicare acute care benefit package. Its enrollees are functionally impaired community residents with nursing home level of need. Most PACE enrollees are Medicare/Medicaid dual eligibles for whom PACE organizations receive monthly capitation payments from both Medicare and Medicaid. In 2003, there were 27 PACE organizations nationwide serving roughly 7,000 beneficiaries. The PACE organizations are required to participate in the HOS-M and some sites tested the Core Outcomes and Comprehensive Assessment–Basic (COCOA-B) instrument as described here. In short, experts use the HOS-M to generate annual adjustments to the PACE organizations' Medicare payments reflecting the functional status of each organization's enrollees. COCOA-B is an assessment process related to ongoing quality monitoring and improvement.
HOS-M
A variant of the Health Outcomes Survey, the HOS-M is a brief mixed-mode (mail with telephone follow-up) survey. It is fielded annually to collect functional status information from enrollees in the PACE and the Dual Eligible Demonstrations in Massachusetts, Minnesota, and Wisconsin. This functional status information is used to calculate frailty factors for Medicare payments. The frailty factor system improves payment accuracy by providing additional payments—on top of Medicare's diagnosis-based payment system—to certain plans that enroll a disproportionate number of beneficiaries with functional impairments. The frailty factors were calibrated based on analyses of the Medicare Current Beneficiary Survey, which includes health and functional status measures linked to Medicare costs. Plan-level frailty factors are calculated on the basis of the HOS-M functional status measures of six ADLs: bathing, dressing, eating, transferring, toileting, and walking. The question reads: "Because of a health or physical problem, do you have any difficulty doing the following tasks without special equipment or the help of another person?" The response options are "No, I do not have difficulty," "Yes I have difficulty," and "I am unable to do this activity." Individuals are coded as having an impairment based on reporting difficulty performing or the inability to perform an activity without special equipment or the help of another person. Researchers have conducted many analyses using these items in the Medicare Current Beneficiary Survey and in analyzing the national HOS used in all Medicare managed care plans.
Because the HOS-M is used for payment, the protocol focuses on achieving high response rates and ensuring representation of highly impaired enrollees. To this end, the instrument is substantially shorter and more targeted than the HOS, and response options for some items are tailored for a highly impaired population. Proxy respondents are encouraged, and family contact information is collected from health plans to assist in reaching proxies if needed. In addition, the survey is available in multiple languages to accommodate diverse populations, and the ADL items appear at the beginning of the instrument. The survey vendor is instructed to focus on getting answers to the ADL items in telephone follow-up—a survey is not considered complete unless the respondent answers all six ADL items (bathing, dressing, transferring, toileting, walking, and eating). Between April and August 2002, RTI International pilot tested the HOS-M protocol in four PACE organizations (Walsh et al., 2003).
HOS-M proxies include family members and health or social service professionals. In 2003, 65% of survey respondents were proxies due to the high level of physical and cognitive impairment in the PACE population. Proxies responded to the HOS-M due to one or more of the following problems: the enrollee's physical health (51%), memory loss or mental health (49%), inability to speak or read English (20%), and other reasons (22%)(Khatutsky et al., 2004). The pattern was similar in subsequent rounds of the HOS-M (Khatutsky, Walsh, & Brown, 2006; Khatutsky, Walsh, Kramer, & Brown, 2005).
COCOA-B
The COCOA-B instrument is used to monitor quality in PACE plans as part of the Outcome-Based Continuous Quality Improvement System. A variant of the Outcome and Assessment Information Set used in home health, COCOA-B is designed specifically for use in PACE organizations. The Outcome and Assessment Information Set is "a group of data elements that represent core items of a comprehensive assessment for an adult home care patient and form the basis for measuring patient outcomes for purposes of outcome-based quality improvement" (http://www.cms.hhs.gov/OASIS). The COCOA-B is an extensive assessment instrument covering a wide range of clinical topics. RNs complete much of the COCOA-B by interviewing PACE participants as part of the assessment process. The instrument also includes clinical evaluations by occupational and physical therapists (OTs and PTs); these clinical evaluations include ADL measures and information about the use of special equipment. The University of Colorado conducted a pilot of the COCOA-B in 13 PACE plans—including the 4 plans participating in the HOS-M pilot—between June and November 2003.
Although the COCOA-B includes these OT and PT evaluations of ADL status, it does not include self- or proxy reports of ADLs. For the purpose of this study, the University of Colorado team incorporated the HOS-M ADL items into the 2003 COCOA-B pilot in two ways. First, RNs conducting the in-person participant interview portion of the assessment asked participants the HOS-M questions and recorded their answers; this provided face-to-face interview results for the six ADL items. Second, these same RNs also recorded their own assessment of each participant's ADLs in response to the same HOS-M questions. These nursing assessments provided a form of proxy reporting for those unable to respond to the face-to-face interview. They also provided another source of ADL ratings for the full sample (i.e., the RN assessment using the HOS-M items) for comparison to the OT or PT evaluations, self-report, and the standard mail with telephone follow-up mode of HOS-M administration. We added the HOS-M items to the COCOA-B pilot to inform the Centers for Medicare & Medicaid Services about the potential impact of various modes of administration on frailty factor calculations.
|
Methods
|
|---|
The objective of our study was to evaluate differences in disability rates resulting from various data collection methods in this sample of PACE enrollees using the same battery of six ADL questions. In addition, we compared these measures between HOS-M self-respondents and proxies. Figure 1 summarizes the various modes of administration used in the study.
View larger version (11K):
[in this window]
[in a new window]
|
Figure 1. Approaches to activity of daily living (ADL) measurement. HOS-M = Health Outcomes Survey–Modified; COCOA-B = Core Outcomes and Comprehensive Assessment–Basic; RN = registered nurse
|
|
The HOS-M items used in both processes (survey and face-to-face interviews or RN assessments) were identical, and we coded individuals as impaired if they reported receiving help, using special equipment, or having difficulty performing an ADL without special equipment or the help of another person. For the clinical evaluations conducted by OTs or PTs, we created a crosswalk that identified individuals as impaired in an ADL if they required any degree of personal assistance or used an assistive device. The crosswalk was imperfect, as the clinical evaluation did not include an option for having difficulty, although one can infer this in evaluating a need for equipment or help.
Sample
The initial sample consisted of the 900 enrollees in four PACE sites who were simultaneously participating in the pilot of HOS-M and a field test of the COCOA-B. From this sample, we limited our findings to two subsets. The first subset included the 547 PACE enrollees for whom we had full ADL data in the HOS-M and the COCOA-B. The second group was a subset of the 547 and included 432 enrollees with full ADL data for whom HOS-M proxy status—whether the survey was completed by the enrollee or a proxy—was defined. We used this smaller sample in comparing the self-respondents and proxy respondents to the HOS-M.
Data Collection
New England Research Institutes collected the HOS-M data between April and August 2003 under a subcontract to RTI International. They included all community-residing Medicare beneficiaries enrolled in the four pilot sites. The University of Colorado research team collected the COCOA-B data between June and October of the same year. The COCOA-B study sample had high levels of physical and cognitive impairments, and 14% of the face-to-face interviews in the COCOA-B field test could not be completed by the sample member, primarily due to cognitive impairment. For these individuals, we had two of the three possible sources of ADL information in the COCOA-B field test: the RN assessment using the HOS-M ADL items and the OT or PT clinical evaluation. For all other individuals, we also received the individual participant's verbal response to the HOS-M ADL questions inserted into the COCOA-B pilot. To provide an equivalent to the HOS-M protocol that permitted proxy responses, we coded either the individual verbal response for the 86% capable of participating or the RN response serving as a proxy for the remaining 14%.
|
Results
|
|---|
Table 1 presents ADL distributions by mode of administration. The sample (N = 547) included all those for whom we had full ADL information from all data sources, regardless of whether the HOS-M responses were provided by the sample members or by proxies. Face-to-face interviews—accepting RN interview assessments as proxies for 14% of the sample—yielded the lowest level of impairment (M = 3.1 ADL impairments), whereas OT or PT evaluations yielded the highest (M = 4.1). Results of the in-person RN assessment and the mail survey with telephone follow-up results—65% of which were completed by proxies—were most similar to each other (Ms = 3.1 and 3.3, respectively). The differences are particularly striking at zero and five to six ADL impairments. The rehabilitation therapists evaluated only 3% of the sample as having zero ADL impairments; in face-to-face interviews, 24% of the sample reported zero ADL impairments, twice as many as indicated in the mixed-mode survey results and the in-person RN assessment. Similarly, the OTs and PTs found almost half the sample to have five to six ADL impairments, whereas only 23% of the sample members themselves reported this level of impairment. The mean number of ADL limitations for study participants collected by face-to-face interview, in-person RN assessments, and OT or PT evaluation were significantly different from the mean number of ADLs collected with the mail survey with telephone follow-up (p <.01). Similarly, the differences in the distributions across individual ADL categories (0 ADLs, 1–2 ADLs, 3–4 ADLs, and 5–6 ADLs) were also significantly different from the mail survey with telephone follow-up (p <.01).
Similar trends were evident in the distributions of individual ADL impairments by mode of administration as presented in Table 2, with all differences highly significant (p <.01). There was little agreement and a wide range of responses across mode of administration for individual ADL impairments, although the discrepancies were not as great for dressing as for the other ADLs. OT or PT evaluations reported the highest levels of impairment on each ADL, whereby almost all study participants (96%) were reported to have difficulty with bathing and 83% were reported to have difficulty with walking across the room.
Table 3 presents the mean number of ADL impairments stratified by HOS-M proxy status for the subset of the study for whom we knew whether the survey was completed by the sample member or a proxy. The HOS-M includes the question "Who completed this survey form?" The response options are "Medicare participant," "family member, relative, or friend of Medicare participant," and "nurse or other health professional." Professional staff serving as proxies are asked to select a response regarding their position, such as "case manager or social worker" or "home health aide or personal care attendant." Not all survey respondents answered this question. As stated earlier, respondents use proxies due to physical or mental health problems, cognitive limitations, inability to speak and/or read English, or for other reasons. It is not surprising, therefore, that HOS-M proxy respondents tend to have higher levels of ADL impairment regardless of mode of administration compared to self-respondents (Khatutsky, Walsh, Moore, & Kramer, 2004). The discrepancies in disability ratings between the various modes of administration were still present and followed the patterns described previously (p <.01).
View this table:
[in this window]
[in a new window]
|
Table 3. Mean Number of ADL Impairments by Proxy Status and Mode of Administration for HOS-M Respondents With Proxy Status Indicated (n = 432).
|
|
In summary, estimates of disability varied substantially by mode of administration in this study. All survey-based approaches yielded lower rates of disability than those derived from clinical assessments. Consistent with previous research, we also found that people reported less impairment in face-to-face interviews than in a mail survey with telephone follow-up. In addition, our findings revealed that clinicians identified more impairment than individuals reported themselves and that clinical measures varied significantly between RNs and OTs or PTs and by instrument (HOS-M vs OT or PT clinical evaluation). Although some of the individual approaches reported here mixed two types of respondents (self and proxy) or modes (mail with telephone follow-up), they represent real-world ways in which such data are collected, and clinical, payment, and service-eligibility decisions are often based on such mixed data collection methods.
|
Discussion
|
|---|
This study has implications for policy, delivery, and practice. First, it is important for clinicians and service providers to note how valuable it is to look beyond self-report in evaluating functional status, as individuals often report fewer functional impairments than do professionals such as rehabilitation therapists, especially in personal interviews. Thus, relying on patient interview may be inadequate for assessment and care planning. Long-term-care eligibility determination, when based on functional status, may also be sensitive to whom is asked about functional status and how that information is gathered. Based on the findings of this study, if the data were collected by in-person interview, about a quarter of individuals would be determined to have no ADL impairments and hence would be ineligible for any long-term-care services based on ADL counts or impairments in specific ADLs. This is in a sample of PACE participants, individuals who have all met the nursing home level-of-care criteria in the states where they reside and who presumably are representative of elderly home- and community-based waiver populations.
Second, researchers evaluating survey data findings from studies that measure disability levels need to take into account mode of administration when making comparisons across different samples. Clearly, different approaches yield different results, reinforcing previous findings about the impact of measurement differences in calculating disability rates, this time using a single sample.
Third, the Centers for Medicare & Medicaid Services currently collects information about functional status through a mail survey with telephone follow-up from which it calculates frailty factors to enhance payment to plans such as PACE that enroll a disproportionately frail population. Alternative approaches (e.g., embedding the questions within the COCOA-B instrument or substituting clinical assessment of functional status) would yield very different functional status distributions and hence very different payments to health plans. The average frailty factor (data not shown) based on these results would vary from 0.3 to 0.7 depending on which approach is used to collect ADL information. Of the survey (i.e., nonclinical assessment) approaches evaluated, the current approach yields the highest payments for health plans. Frailty factors used for risk adjusting payment are calibrated based on analysis of the in-person Medicare Current Beneficiary Survey. Given our findings that people report lower levels of impairment in in-person interviews, the frailty factors calculated from the HOS-M may be generous. Use of clinical assessment data would result in higher payments but would be inconsistent with the underlying payment calibration used.
|
Footnotes
|
|---|
The authors would like to thank members of the PACE OBCQI Team at the Center for Health Services Research (CHSR), University of Colorado, that included P. Shaughnessy, N. Donelan-McCall, M. Kaehny, and A. Kramer, for sharing data with us. 
This project was funded by the Centers for Medicare & Medicaid Services under Contract No. 500-00-0030, T.O.#3. The statements contained in this article are solely those of the authors and do not necessarily reflect the views or policies of CMS. The authors assume responsibility for the accuracy of the information contained in this article.
1 RTI International, Waltham, MA.
Decision Editor: Linda S. Noelker, PhD
Received for publication July 28, 2006.
Accepted for publication March 27, 2007.
|
References
|
|---|
- Dillman, D. A., & Christian, L. M., (2005). Survey mode as a source of instability in responses across surveys. Field Methods, 17 30-52.[Abstract]
- Gruenberg, L., Kaganova, E., & Corazzini-Gomez, K., (2001). Research study of risk adjustment and geographic variation issues in both Medicare and non Medicare cost and use. Final report for the Health Care Financing Administration (HCFA). Cambridge, MA: Data Chron Health Systems, Inc.
- Gruenberg, L., Tompkins, C., & Porell, F., (1989). The health status and utilization patterns of the elderly: Implications for setting Medicare payments to HMOs. Advances in Health Economics and Health Services Research, 10 41-73.[Medline]
- Jette, A. M., (1994). How measurement techniques influence estimates of disability in older populations. Social Science & Medicine, 38 937-942.[Medline]
- Jones, C. A., & Feeny, D. H., (2005). Agreement between patient and proxy responses of health-related quality of life after hip fracture. Journal of the American Geriatrics Society, 53 1227-1233.[Medline]
- Kautter, J., & Pope, G. C., (2004). CMS frailty adjustment model. Health Care Financing Review, 26 (2), 1-20.
- Khatutsky, G., Walsh, E. G., & Brown, P., (2006, January). The 2005 Health Outcomes Survey-Modified (HOS-M) results. Final Tables. Final Report for the Centers for Medicare & Medicaid Services, contract number 500-00-0030, T.O. #3 and 500-00-0024, T.O.#11, RTI International, Waltham, MA.
- Khatutsky, G., Walsh, E. G., Kramer, C., & Brown, P., (2005, September). The 2004 PACE health survey: Methodology and results. Final Report for the Centers for Medicare & Medicaid Services, contract number 500-00-0030, T.O. #3, RTI International, Waltham, MA.
- Khatutsky, G., Walsh, E. G., Moore, A. B., & Kramer, C., (2004, December). The 2003 PACE health survey: Methodology and results. Final Report for the Centers for Medicare & Medicaid Services, contract number 500-99-0264, RTI International, Waltham, MA.
- LaPlante, M. P., (1991). The demographics of disability. Milbank Quarterly, 69 (Suppl. 1/2), 55-77.[Medline]
- McDowell, I., (2006). Measuring health: A guide to rating scales and questionnaires (3rd ed.). New York: Oxford University Press.
- McHorney, C. A., Kosinski, M., & Ware, J. E., Jr. (1994). Comparisons of the costs and quality of norms for the SF-36 health survey collected by mail versus telephone interview: Results from a national survey. Medical Care, 32 551-567.[Medline]
- Ostbye, T., Tyas, S., McDowell, I., & Koval, J., (1997). Reported activities of daily living: Agreement between elderly subjects with and without dementia and their caregivers. Age and Ageing, 26 99-106.[Abstract/Free Full Text]
- Pope, G. C., Adamache, K. W., Walsh, E. G., & Khandker, R., (1998). Evaluating alternative risk adjusters for Medicare. Health Care Financing Review, 20 (2), 109-129.
- Todorov, A., & Kirchner, C., (2000). Bias in proxies' reports of disability: Data from the National Health Interview Survey on disability. American Journal of Public Health, 90 1248-1253.[Abstract/Free Full Text]
- Walsh, E. G., Nason, C. A., Moore A., & Khatutsky, G., (2003, July). Pilot test of the Medicare health survey for PACE and EverCare. Final Report for the Centers for Medicare & Medicaid Services, contract number 500-00-0030, T.O. #3, RTI International, Waltham, MA.
- Weissert, W. G., & Musliner, M. C., (1992). Case-mix adjusted nursing-home reimbursement: A critical review of the evidence. Milbank Quarterly, 70 455-490.[Medline]
TOP
Abstract
PACE
