| HOME | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|---|
| ||||||||||||||||||||||||||||||||
Correspondence: Address correspondence to Brant E. Fries, PhD, Institute of Gerontology, University of Michigan, 300 North Ingalls, Ann Arbor, MI 48109-2007. E-mail: bfries{at}umich.edu
 |
Abstract |
|---|
 TOP Abstract BackgroundMethodsResultsDiscussionConclusionReferences |
|---|
Key Words: Long-term care • Preadmission screening • Eligibility • Nursing home • Home care • Assessment • MDS-HC
|
Background |
|---|
|
TOPAbstractBackgroundMethodsResultsDiscussionConclusionReferences |
|---|
The research showed mixed results concerning the utility and costs of preadmission screening in both home care and nursing home populations. Instruments designed specifically for preadmission screening are often "homegrown" assessment tools, formed around specific policy agendas, and have been shown to be unable to identify accurately the appropriate population (Fries, Shugarman, Morris, Simon, & James, 2002; Harrington & Curtis, 1996; Jackson, Eichorn, & Blackman, 1992; Spector & Kemper, 1994). There is, however, agreement on the need for screening instruments that are well planned, targeted to particular subsets of service seekers, and supported by empirical research (Brummel-Smith, Boult, Boult, & Pacala, 1998; Harrington & Curtis, 1996; Iverson & Polich, 1987; Jackson, Eichorn, Sokoloff, & VanTassel, 1993; Leutz, Abrahams, & Capitman, 1993; Pepe, Applebaum, Straker, & Mehdizadeh, 1997; Yeatts et al., 1987).
In addition to concerns about the content of preadmission screening instruments, there also has been a concern about the relative effectiveness of the modes used by screening programs to gain information about an individual's health, cognition, and functional status. Researchers in a variety of disciplines have investigated the comparable effectiveness of self-reports, telephone interviews, and in-person interviews to collect health information (Coleman et al., 1998; Herzog & Rodgers, 1988; McAuliffe, Geller, LaBrie, Paletz, & Fournier, 1998; O'Toole, Battistutta, Long, & Crouch, 1986; Pless & Miller, 1979; Rissel & Ward, 1999; Sharkey & Haines, 2001; Siemiatycke, Campbell, Richardson, & Aubert, 1984). While such research generally has indicated the results obtained from both telephone and mail interviews are comparable to in-person interviews in terms of quality, these findings may be of questionable applicability to preadmission screening, where the individual is actively seeking services and thus has an incentive to give biased responses. The National Long-Term Care Channeling Demonstration did utilize a number of interview modes to target eligible individuals (Applebaum, Baxter, Callahan, & Day, 1985; Applebaum & Wilson, 1987; Kane, 1988). In a Channeling study comparing telephone screens to assessments, Applebaum and Wilson found that "a systematic telephone screen is reasonably good at identifying those who would be eligible based on a full assessment" (p. 85). They added that the screen offered "several advantages to community-based care programs" when "selection criteria are clearly defined in line with a program's goals." They concluded that "agencies electing to utilize a systematic screen would need to develop quality assurance mechanisms to periodically assess a sample of those persons deemed ineligible at screen" (pp. 85–86).
The Michigan Screening Process and the Level-of-Care (LOC) Algorithm
Michigan's MI Choice initiative is comprised of two statewide community-based long-term care programs targeted at people aged 18 or older at high risk of nursing home placement: the Medicaid Home and Community-Based Services Waiver for Elderly and Disabled Persons and the state-funded Care Management program. Each program is managed by separate agencies housed within the Michigan Department of Community Health (the Medical Services Administration and the Michigan Office of Services to the Aging, respectively). For administrative efficiency, and to enable comparisons among the populations served, both programs use the same screening and assessment tools and procedures. Twenty-three organized health care delivery systems, including Area Agencies on Aging, have contracts with the department as waiver agents and/or care management program providers. Among their many responsibilities, the programs undertake telephone and in-person screening, typically by trained nonprofessional staff, provide care management services by professional nurses and/or social workers, and purchase and arrange services for eligible participants from enrolled provider agencies.
Michigan relies on a standardized preadmission screening process to determine the medical eligibility of those seeking MI Choice services. Unlike some screening models (Curtis & Harrington, 1999, Tonner, LeBlanc & Harrington, 2001), the design implemented by Michigan does not use either a single point of entry or a separate entity to carry out preadmission screening. Instead, the state has granted that authority to the individual programs. Potential participants (or others on their behalf) initiate the enrollment process by calling one of the programs. During this telephone call, the individual or surrogate is questioned about information that allows a telephone screener to complete a 32-item screen. The items forming the screen are a subset of the more comprehensive MI Choice assessment instrument, into which is embedded the Minimum Data Set for Home Care (MDS-HC), a comprehensive assessment system compatible with the nursing home MDS (Morris et al., 1997).
Using the information gathered from the telephone screen, a decision is made whether it is likely that the individual is medically eligible to receive home care services. This decision is informed by the MI Choice Level-of-Care algorithm. For those who score into certain algorithm categories, a nurse and/or social worker conducts an in-person assessment in the person's home. Afterward, the algorithm is reapplied to the data obtained in person, and a final determination of medical eligibility is made.
The level-of-care algorithm is the result of research to define different levels of service need for potential participants, based on their clinical, social, and functional characteristics (Fries et al., 2002). It groups people into five categories corresponding roughly to the type of services one would expect to be provided, as shown in Table 1.
|
The primary difference between the telephone screen and the in-person assessment results directly from the limitations inherent in telephone contact. Over the telephone, the interviewer is limited to direct responses (albeit with probes for additional clarification); there is often no corroborating evidence available from seeing the person, their home, or others, such as informal caregivers. The screener must rely on the truthfulness and the accuracy of the respondent, who could be the individual seeking services, a family member, a service provider, or some other person. The in-person assessment, on the other hand, can be informed by all such sources of information, with the assessor determining the most appropriate response when sources differ. The purpose of this study was to explore the consistency between responses obtained through the telephone screen and responses obtained during the in-person assessment, to examine whether particular issues affect the accuracy of the matched responses, and to determine if the cost of the telephone screen merited the investment.
Issues of Interest
The following four specific issues were identified for exploration:
|
Methods |
|---|
|
TOPAbstractBackgroundMethodsResultsDiscussionConclusionReferences |
|---|
We compared the screen's accuracy on three dimensions: the accuracy of the "likely eligible" versus "likely not eligible" findings; the comparability of the assigned level of care; and the match between individual responses to identical items on the telephone screens and the in-person assessments. The measures of comparison used to evaluate agreement between the telephone screen and the in-person assessment were the direct agreement between the two assessments and the kappa statistic. The kappa statistic measures the strength of agreement between multilevel variables, adjusting for chance fit. For 2 x 2 tables we used the simple kappa statistic, while for larger tables we used the weighted kappa, employing Fleiss-Cohen weights as computed by the SAS statistical package for PCs, Version 8e; 95% confidence intervals are provided as well. Kappa statistics range from approximately 0 to 1.0; values in excess of.4 are considered to reflect moderate agreement; values above.8 reflect excellent level of agreement (Fleiss, 1981). When there was a disagreement, we also considered whether the telephone screen assessed the potential participant in a more or less resource-intensive level-of-care category. To accomplish this comparison, we considered only those assessments for which there was a disagreement with the screen and tested whether one percentage significantly exceeded 50%, using the sign test. To address the questions raised earlier about differences in agreement based on the time between assessments, type of caller, and other factors, we tested the difference between the kappas at the 5% level of significance.
|
Results |
|---|
|
TOPAbstractBackgroundMethodsResultsDiscussionConclusionReferences |
|---|
|
The value below the bolded diagonal in Table 2 represents individuals who, at screening, scored as ineligible but who, with the information collected at the in-person assessment, were subsequently scored eligible. Six percent of the matched assessments fell into this group. Assuming that screeners strictly abided by the protocols of the screening process and did not offer any of these people an in-person assessment, then 1,460 people would have been wrongly excluded by the telephone screen. This subgroup represents the screening "false negatives."
In contrast, the value above the diagonal on Table 2—6,381 individuals, or 27% of all program seekers—represents those who scored as eligible at screening, but who were subsequently scored not eligible at the in-person assessment. This subgroup represents the "false positives" of the screen.
While a major interest of the study was to understand the accuracy of the eligibility determination, we also sought to test the accuracy with which individuals were assigned to the specific categories by the screening algorithm. The relationship of the assignments to the five levels of care, as identified by both the telephone screens (columns) and the in-person assessment (rows), is shown in Table 3.
|
Next, we compared each screen question to its corresponding in-person assessment item to evaluate whether specific questions or types of questions matched and, when they did not, whether these differences were responsible for the level-of-care mismatches. Table 4 presents data on the agreement of comparable questions on the telephone screen and items on the assessment. Note that in several instances different thresholds for the same issue are used to calculate several levels of care. For example, information about bathing activities is used in the screening algorithm to identify three different levels of care: twice, we make a dichotomy between totally independent individuals and all others (for the Nursing Home and Information and Referral levels of care), and once, we split out those individuals who, at most, need supervision (for the Intermittent Personal Care level of care). Table 4 reports the magnitude of errors, that is, the percentage of the sample for which the information gained over the telephone on a particular screen question represents a more disabled answer (screen overreport) or a less disabled answer (screen underreport), when compared to the information gained in person. The larger of these two percentages for each question is bolded. Save for the question on dressing, all screen questions had differences that were significant at p <.01.
|
To test whether certain issues might be contributing consistently to level-of-care mismatches, we ran a series of analyses, in each one correcting mismatched responses on a single screen question and recalculating the level-of-care algorithm. We were particularly concerned with screen questions that agency staff considered problematic when the screen was first designed (e.g., RN Monitoring). The results of these analyses suggest that no particular question is responsible for a large share of level-of-care mismatches (results not shown).
The remainder of our analyses considered whether there were any of several intervening variables that could explain the differences between the telephone and in-person assessments. In each, we considered the effect on the weighted kappa statistic, computed on the 5 x 5 tables of level-of-care assignments, similar to those in Table 3. The first considered the source of telephone screen information. Various people may provide information for the telephone screen: the potential participant (28% of callers); a friend, neighbor, child, spouse, or other relative (53%); or a nurse, social worker, home health aide, or legal guardian (19%). Overall, comparing across these three caller groups, health professionals appear to provide the most accurate information (
= 0.28; CI =.24,.32), and self-reporters provide the least accurate information ( =.19; CI =.16,.22), with the difference statistically significant (p =.001), although the kappa statistics for all three groups were marginal. All three sources were much more likely to report the individual on the screen as more dysfunctional by similar margins.
Next, we examined whether the location of the potential client at assessment affected screen/assessment agreement. Out of the 23,595 individuals with both screen and assessment information, 93% were assessed in their own homes. Other potential participants were assessed in hospitals, nursing homes, or other settings. There were no significant differences among the kappa statistics for individuals living in their own homes versus individuals in institutional or other settings (results not shown).
A third possible reason for poor agreement might be the span of time between screen and assessment. Many people (or their surrogates) seek help during a crisis when they are feeling desperate for support or relief; it is possible that such situations resolve to some degree by the time an in-person assessment is carried out, or a new problem could arise. We hypothesized that the longer the time period between the telephone screen and the in-person assessment, the less likely the level of care would match. We compared kappa statistics for people who were assessed within 5 days of their screen versus those assessed after 5 days, and replicated these analyses for 7-day and 14-day splits. In all cases, the differences in kappa statistics were negligible and not statistically significant at the 5% level.
Fourth, we tested whether there was a temporal change in accuracy of the match between the screen and the in-person assessment over the course of the program. In particular, we examined whether accuracy was greater before or after early June 2000, when training of all screeners was completed. At the time that the screening instrument was introduced to the field, the state contracted to provide training and technical assistance to screeners. Using feedback from screening staff, the state and its information technology contractor, in consultation with the research team, amended the design of the screening form several times, rewording MDS-HC items into questions, and providing additional instructions for scoring answers. One might expect that accuracy was greatest during training or, alternately, that it improved as users gained experience with its use. About 46% of the assessments until August 2002 were performed before June 2000. Contrary to expectations, we found no significant difference in kappa statistics pretraining and posttraining.
Finally, we examined whether particular agencies had significantly better rates of agreement between the telephone screen and in-person assessment. Out of 23 agencies, two had kappa statistics demonstrating moderate fit, with the highest at.47. However, with a 95% confidence interval of this latter agency ranging from.39 to.56, this kappa statistic was not significantly different than the acceptable value of.40. There was no reason to expect, a priori, that this agency was performing better than others. Most agencies, when there was a lack of agreement between the assignments based on the screen and the in-person assessment, reported more disabled levels of care at a rate of at least three to one; one agency had almost equal assignments (51% more disabled levels of care).
Cost Effectiveness of Telephone Screening Activities
The primary purpose of the MI Choice telephone screen is to identify those potential participants who will be unlikely to meet the medical eligibility threshold. In Michigan, the state-funded Care Management program has used telephone screening since the program was established in 1983. The face validity of this effort seems obvious: In rural areas with large driving distances, or in congested urban areas, a home visit by a nursing or social work professional can be very time consuming. In contrast, trained staff can conduct the telephone screen in about 20 minutes, and a clinical degree is not necessary. The new telephone screen discussed here laid a scientific basis for its content and the decision making that resulted from its use.
To measure the cost of the MI Choice screen, we gathered data about the staff costs associated with screening and assessment activities. Although wages and procedures vary substantially across the state, we estimate that the average telephone screen costs about $3.35 in direct staff costs to administer (2001 estimates). Screeners generally are required to have a high school education. In comparison, most agencies use either a registered nurse or a nurse/social worker team to assess prescreened people in their homes. Agencies report that the in-person assessment takes 1–2 hours, but driving times can add several additional hours. Not including this highly variable driving time, an in-person assessment costs a minimum of $30–$70 in direct staff costs (2001 estimates).
Without telephone screening, all callers would receive an in-person assessment. If there are 100 callers and a cost of $70 per assessment, the cost to provide in-person assessments to all callers would be $7,000. With telephone screening, of the 100 callers, all would get a telephone screen (for a cost of $335), but only 84 would be assigned to the likely eligible groups (Nursing Home, Home Care) by the screen and thus receive an in-person assessment (for a cost of 84 x $70 = $5,880). Thus the telephone-screening model would cost $6,215 ($335 + $5,880), for a savings of 11% in direct staff time over the cost without screening.
|
Discussion |
|---|
|
TOPAbstractBackgroundMethodsResultsDiscussionConclusionReferences |
|---|
This analysis confirmed that the responses on the telephone screen generally portrayed potential participants as needing heavier care than did the direct observations of assessors. In fact, on two thirds of the telephone-screen questions, the more common mismatch was an overestimation of illness and dependency. We could discern no trend in particular issues or topics that might explain this direction. This finding about exaggerated self-reporting is interesting, as it runs counter to a widely held belief among program staff that older adults typically underreport their circumstances when they call for themselves, as they fear a nursing home placement if they are found too dependent to stay at home. This did not appear in our data; self-reporters were less accurate and more likely to overreport the severity of disabilities across the continuum of levels of care. The fact that exaggeration characterizes two thirds of all the telephone-screen responses, coupled with the finding that no particular type of respondent or type of question accounted disproportionately for mismatches, suggests that well-intended overstatements are the rule rather than the exception.
Despite longstanding concerns that individuals in hospital or nursing home settings could not be screened or assessed properly when they were acutely ill or recovering from illness, there were no significant differences among the kappa statistics for people living in their own homes versus people in institutional or other settings. Neither did the time between screen and assessment explain mismatched level-of-care scores. These findings suggest that location and timing do not affect the fit between screen and assessment information. We also did not find a difference between earlier and later cases in the program. Many factors were likely confounded here, including training (and detraining) effects, turnover in screening or assessment staff, and potential participants (or their surrogates, including hospital discharge planners) figuring out the screening system and altering responses to maximize the individuals' chances of receiving services. There were also changes in the funding and availability of waiver slots in the more recent past that could have had unexpected effects.
We did identify a few agency-level effects. Although the overall fit by agency was poor, 2 agencies among 23 had moderate kappa statistics, and a third agency demonstrated no overreporting. While it was beyond the scope of this study to identify the specific agency practices or training that may contribute to differences in performance, these findings hold out the hope that telephone screening can be performed acceptably. Such an analysis would be a useful next step to identify practices that could render the telephone screen more accurate across agencies.
It is important to note that there are several challenges to the generalizability of these results. First, they represent the results of a single—although statewide—program, which may not be representative of other home care programs or other states' waiver programs. Second, there were marked changes in the programs during the time of this study, including substantial contraction of home care slots. Thus, although this trend has been reversed, its effect on these results is unclear. Third, the results are highly contingent upon the specific content and format of the screen. Although the MDS-HC items have demonstrated reliabilities, there has not been systematic reliability or validity testing of the screen items that attempted to turn MDS-HC items into questions that could be asked over the telephone. Finally, we did not have data on the 14% of the sample representing individuals who were screened but did not go on to a full assessment. While we found substantial up-coding on the screen for most individuals, this could not occur in these cases which were, perforce, coded on the screen at the lowest levels of impairment. Thus, complete data on these individuals might have positively affected our match rates.
|
Conclusion |
|---|
|
TOPAbstractBackgroundMethodsResultsDiscussionConclusionReferences |
|---|
Our findings also indicate that no particular items on the screen are responsible for mismatches and that telephone answers are consistently "more impaired" than assessment answers. These two findings suggest that exaggeration is probably responsible for many inaccuracies. Unfortunately, no telephone tool can easily overcome this problem, although there may be lessons to be learned from agencies that achieve more consistent results. We also discovered that certain callers are marginally better sources of information than are others. Therefore, obtaining information from health professionals or other corroborating sources may improve the screen's ability to predict level of care and eligibility and should be weighed against potential costs.
However, in the end, these findings also echo a familiar cautionary tale: You get what you pay for. Our analysis did not support the use of telephone screening as a direct substitute for information gained from in-person assessment. Our analysis did support the use of telephone screening as a cost-effective mechanism to screen out the least impaired service seekers as well as to identify those for whom a full assessment is warranted. Although in-person assessment is critical to obtain an accurate picture of an individual for care-planning purposes, a low-cost screening mechanism is highly desirable in order to eliminate clearly ineligible applicants. Given their recent budget woes, many states may seek more efficient methods to identify those likely eligible for home- and community-based and nursing home services. Michigan's experience provides a timely lesson in the utility and limits of telephone screening to achieve such results.
|
Footnotes |
|---|
1 Institute of Gerontology, The University of Michigan, Ann Arbor.
2 School of Public Health, The University of Michigan, Ann Arbor.
3 Ann Arbor VA Medical Center, MI.
4 RAND Corporation, Santa Monica, CA.
5 Research and Training Institute, Hebrew Rehabilitation Center for Aged, Boston, MA.
Decision Editor: Linda S. Noelker, PhD
Received for publication April 2, 2003. Accepted for publication April 20, 2004.
|
References |
|---|
|
TOPAbstractBackgroundMethodsResultsDiscussionConclusionReferences |
|---|
This article has been cited by other articles:
 |
|
 |
|
  S. Hedrick, M. Guihan, M. Chapko, L. Manheim, J. Sullivan, M. Thomas, S. Barry, and A. Zhou Characteristics of Residents and Providers in the Assisted Living Pilot Program Gerontologist, June 1, 2007; 47(3): 365 - 377. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
C. Hawes, B. E. Fries, M. L. James, and M. Guihan Prospects and Pitfalls: Use of the RAI-HC Assessment by the Department of Veterans Affairs for Home Care Clients Gerontologist, June 1, 2007; 47(3): 378 - 387. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||
| HOME | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|---|
