| HOME | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|---|
| ||||||||||||||||||||||||||||||||
Correspondence: Address correspondence to Timothy J. Lowe, MSW, LSW, LNHA, Rutgers University, Institute for Health, Health Care Policy, and Aging Research, 30 College Avenue, New Brunswick, NJ 08901-1293. E-mail: Lowe_Timothy{at}Yahoo.com
 |
Abstract |
|---|
 TOP Abstract MethodsResultsDiscussionReferences |
|---|
Key Words: State policy • Quality monitoring • Nursing home quality • Long-term care outcomes • Consumer information
Compared with the managed care and hospital industries, the long-term care industry has been slow to adopt CQI processes and consumer satisfaction measures (Applebaum et al., 2000; Spuck, 1992). This may be due, in part, to the difficulty of incorporating such techniques in an environment dominated by small, for-profit providers serving a diverse and frequently cognitively impaired population. Yet in the wake of several federal reports describing quality problems in U.S. nursing homes and assisted living facilities (e.g., U.S. Government Accounting Office [GAO], 1997, 1999; IOM, 2001), and in a changing legislative environment mandating the documentation of outcomes of programs and budget expenditures (e.g., Government Performance and Results Act, 1993; National Nursing Home Quality Initiative, 2002 [see U.S. Department of Health and Human Services, 2002]), several states and long-term care associations have now begun to develop satisfaction measurement initiatives. Currently, state legislation is being written, survey instruments designed, and public funds channeled to include consumer satisfaction data in institutional long-term care licensure and certification inspections and consumer information systems (Soberon-Ferrar, 2000). To our knowledge, however, no published studies to date have provided a comprehensive review of state satisfaction initiatives in nursing homes and assisted living facilities. The present study was undertaken for this purpose.
Background
Although there is no consensus within the health care industry on the role consumer satisfaction information should play in the assessment of quality of care (Bliesmer & Earle, 1993; Krowinski & Steiber, 1996), researchers and policymakers believe its role to be crucial (Davis, 1991; Harrington, 1991; Kruzich, Clinton, & Kelber, 1992). Donabedian (1966), for example, has stated that "achieving and producing health and satisfaction, as defined for its individual members by a particular society or subculture, is the ultimate validator of the quality of care" (p. 166).
Although most research on consumer satisfaction has taken place in hospitals and in industry, there are a small number of research efforts that have been conducted in nursing homes (NHs) and assisted living facilities (ALFs). Several conclude that consumer satisfaction is an important indicator of quality of care in both settings, despite the philosophical differences between NHs and ALFs in the areas of choice, dignity, privacy, independence, individualization, and normal lifestyles (Gesell, 2001; Pearson, Hocking, Mott, & Riggs, 1993; Uman & Urman, 1997; Kane & Wilson, 1993). Other studies in NHs have concluded that resident satisfaction data provide information about the quality of care that is different from information gleaned from clinical indicators, family members, NH staff, or regulatory practices and evaluations (Berlowitz, Du, Kazis, & Lewis, 1993; Lavizzo-Mourey, Zinn, & Taylor, 1992; Uman & Urman, 1997). Reported findings from a national study of ALFs indicate that satisfaction can have far-reaching consequences for resident and family behavior. For example, in 24.1% of the study's cases, dissatisfaction with quality of care led to residents' choosing to depart from a particular ALF (Phillips, Hawes, Spry, & Rose, 2000).
It should be noted that research studies of consumer satisfaction with long-term care services are currently in a nascent stage of development. Although several instruments have been developed, the lack of standardization in item content, format, and overall instrument design and the reliance on small studies or one-facility sampling indicate the need for research into the most appropriate and cost-effective methods of measuring satisfaction. Best practices have yet to be identified. Additionally, there is no generally accepted theoretical model to explain why some individuals are satisfied whereas others are dissatisfied with a given level of care (Applebaum et al., 2000); thus, the meaning of "consumer satisfaction" not only varies across studies but appears to have characteristics both of a unidimensional construct (one that can be measured globally) and a multidimensional construct (one that requires multiple items or subscales; M. A. Smith, 2000). This may be due to the intertwining of satisfaction with care and perceived quality of life (Kane, 2001). Further ambiguity exists over the definition of who is the "consumer" in long-term care (the resident, the family, or both) and the manner in which different consumers' needs and concerns should be measured and compared (Levin, 2001). These ambiguities complicate state efforts to assess and monitor consumer satisfaction. Moreover, there is controversy over whether consumers can evaluate the technical and structural dimensions of care, and thus these dimensions will still require evaluation by professionals (M. A. Smith, 2000). A national study of ALFs has demonstrated the importance of including both consumer satisfaction measures and professional measures of structure and process for performance evaluation in residential care (Hawes, Phillips, & Rose, 2000; Phillips et al., 2000).
Why Initiate Satisfaction Measures?
With the Nursing Home Consumer Assessment of Health Plans (NHCAHPS) initiative, the Centers for Medicare and Medicaid Services (CMS) is moving to expand information on quality of care that is available to consumers and purchasers making choices among NHs (Agency for Healthcare Research and Quality, 2001). One goal of the initiative is to develop a survey instrument that can be used nationwide to measure consumer satisfaction with the quality of NH care. A second goal is to standardize the reporting of satisfaction data as an additional quality measure for NHs. Similar initiatives for ALFs and home care services are currently planned.
Many unknowns remain as to how such an intricate system would be implemented in all states and how the many logistical, methodological, and cost challenges entailed in creating and disseminating a valid and reliable measure of consumer satisfaction would be addressed in such a national system. Given the complexity and uncertainty of this effort, implementing and evaluating a series of state-by-state initiatives may be the best way to develop an experience base to determine the most satisfactory approaches to systemwide consumer satisfaction measurement in long-term care.
Several tangible benefits might result from initiating resident satisfaction measures pro-actively at the state level. One benefit is value purchasing. Satisfaction measurement plays a key role in the determination of health care "value" (Ashley & Strasser, 1997). As the cost of health care services continues to rise, purchasers are seeking to measure the "return" they are receiving on the billions of dollars they are spending. This has brought the concept of quality, as measured by consumer satisfaction, to the forefront of health care (Friedman, 1995; Koenig & Kleinsorge, 1994). The availability of trustworthy satisfaction data could help to drive marketplace competition for better-quality care (Harrington, 1991; Zinn, 1994).
A second benefit is the facilitation of consumer choice. Selecting a long-term care facility is a difficult venture (Harrington, 1991; G. P. Smith, 1996). Disseminating current, accurate, and easy to understand reports on NH and ALF satisfaction would not only serve to assist elders and family members in the selection of quality facilities through "informed choice" but would also potentially increase facility competition through market pressures (Zinn, 1994).
A third benefit is propelling CQI initiatives by facilities and states. In an environment in which publicly available satisfaction information provides the foundation for better value purchasing and consumer choice, the feedback provided by such measures to facility management can drive efforts to improve performance in areas where problems are identified (Reinhard et al., 1999). Likewise, states might use satisfaction results to develop statewide quality improvement strategies (Rantz et al., 2001).
|
Methods |
|---|
|
TOPAbstractMethodsResultsDiscussionReferences |
|---|
The questionnaire addressed the following areas: use of satisfaction surveys in NHs or ALFs, person(s) to contact about instrument development and implementation issues, how long the instrument has been in use, frequency and method of administration, and how the information is used. If applicable, respondents were asked to provide a copy of their satisfaction instrument.
In all cases, if a survey was returned indicating the use of a satisfaction measurement or had missing information, the state director was contacted for clarification. For example, several states reported satisfaction instrument use in NHs, but failed to report if they also surveyed ALFs.
|
Results |
|---|
|
TOPAbstractMethodsResultsDiscussionReferences |
|---|
|
It should be noted that two additional states reported a regulatory interest in initiating satisfaction measures: Maine and Massachusetts. In 2002, Massachusetts initiated a project to develop a satisfaction instrument for families of NH residents. Further, several other states that have not implemented consumer initiatives at the state level have provider or consumer organizations that collect and publish data on NH quality, including satisfaction measures. For example, the Health Care Association of Michigan, which is an NH provider organization, has contracted with a marketing firm to produce a consumer guide and Web site that includes satisfaction scores for Michigan residents. Similar consumer information efforts have been undertaken by provider groups in Alabama, Indiana, Ohio, and West Virginia, as well as by consumer groups in California (California Advocates for Nursing Home Reform) and Washington (Citizens for Improvement of Nursing Homes). In addition, several proprietary firms have developed satisfaction instruments and have marketed them to states or individual health care corporations (e.g., ServiceTRAC, Inc., and Vital Research, LLC). As these are consumer, proprietary, or provider-initiated programs and not state initiatives, these efforts are not described here.
Instrument Administration
As can be observed from Table 1, of the seven states reporting continuous measurement of resident satisfaction data, the majority (n = 6) opted for an annual collection schedule. The exception in this group was Iowa, which reported a variable administration schedule, ranging from 1 to 2 years, although Iowa's regulations, now amended, "originally allowed for a one to three year collection cycle" (A. Martin, personal communication, Iowa Division of Elder Affairs, June 13, 2001).
Four of the six states actively engaged in ongoing resident satisfaction measurement efforts reported by using a face-to-face data collection technique. The exceptions to this were Iowa and Oregon, which used mailed instruments exclusively, and Vermont, which used both a mailed instrument and telephone interviews. It should be noted that Iowa only collects data from ALFs, whereas Oregon uses the same instrument regardless of the long-term care setting type.
Response rates from the states reporting use of satisfaction measures also varied, ranging from 51% (Iowa and Oregon) to 87% (Ohio). On the basis of a statistical analysis, Iowa has set the minimum required number of completed instruments per facility at 30. Interestingly, Florida and New Jersey have selected the same figure. Obviously, the goal set for response rates is also related to cost.
The development and administration of family satisfaction instruments was reported in four states (Florida, Iowa, Montana, and Ohio), although only Florida, Iowa, and Ohio reported that they are continuing the satisfaction measurement process. New Jersey reported a desire to develop a family satisfaction instrument; Texas, however, uses family members only as proxies for cognitively impaired residents unable to complete a satisfaction measure. Massachusetts, as noted, is developing a family satisfaction instrument.
Only Florida, New Jersey, Ohio, and Texas reported using a separate cognitive screen at the time of instrument administration for determining resident ability to participate in the measurement process. Florida adapted a modified version of the Mini-Mental State Examination (MMSE; Folstein, Folstein, & McHugh, 1975). New Jersey examined several screening instruments and adopted the Minimum Data Set Cognition Scale (MDS-COGS; Hartmaier, Sloane, Guess, & Klock, 1994). Ohio developed its own screening instrument from a previous clinical study (Ejaz, Jones, & Rose, 1994). Oregon has designed a client assessment system (known as the Client Assessment Planning System) that relies on case manager evaluations of residents' behavioral status. Residents who receive a poor behavioral evaluation, as judged by the case manager, are excluded as cognitively impaired from the satisfaction assessment. Texas opted to use the Short Portable Mental Status Questionnaire (Pfeiffer, 1975) and leaves the inclusion or exclusion of residents to the surveyor's clinical judgment.
Instrument Design
A review of published, industry-designed, and proprietary resident satisfaction instruments shows wide variation in the number of items included (Castle, Lowe, Lucas, Robinson, & Crystal, in press). A similar variation was found in the 10 instruments provided by states that were or had been engaged in satisfaction measurement (see Table 2). The number of items included ranged from 7 to 70, with a mean of 25. Texas and Ohio, both utilizing face-to-face interviews, reported using 13-item and 49-item resident satisfaction instruments, respectively. The exception in this group was Iowa, which used a mailed instrument containing 70 items for residents.
|
Domains of content were specified in 5 instruments (50%) and ranged in number from 6 to 11 (M = 8). With the use of qualitative analysis procedures, items from each of the instruments were classified into like areas so that they could be compared. For example, food from 1 instrument and food services from another instrument were both classified as meal domains. From this analysis, 12 fundamental domains of resident satisfaction were identified through examination of 227 items contained in the sample of 10 existing instruments. The domains identified through this process include activities, administration, care, environment, independence or choice, meals, restraints (unique to Texas), services or amenities, staff, toileting (unique to Texas), overall satisfaction, and well-being. Unique to the ALF instrument used by Iowa were the domains of community and security.
As shown in Table 2, fewer than half (n = 4) of the states reporting use of resident satisfaction measures evaluated their instruments for reliability and validity. Iowa, New Jersey, and Ohio provided detailed reports of their respective instrument development and testing. Information on the development of Iowa's instrument and administration was obtained from the report of the 1999 National Assisted Living Resident Satisfaction Study (Assisted Living Federation of America [ALFA], 1999).
Cost per Resident Interview
Cost data obtained from a literature review of survey research methods, from state program respondents involved in the implementation of measurement systems in Florida, Iowa, and Ohio, and from several researchers (Gwen Uman of Vital Research, LLC, and Margaret Wylde of ProMatura Group, LLC) yielded a range of $5 to $50 per resident for actual or projected satisfaction measurement cost (see Table 3). These reports indicate that conducting face-to-face interviews by using an outside vendor entails roughly two to four times the cost of mailed instruments. A pilot study of the "Rutgers Satisfaction Assessment Tool–Nursing Home Resident Instrument" in New Jersey is being implemented and will help to determine the cost of face-to-face administration by state personnel during the annual on-site licensure and Medicare–Medicaid certification inspection.
|
Florida
Legislation
Under a law enacted in 1999 and amended in 2000 (Title XXIX, Ch. 400.0225), the Health Quality Assurance Division, Florida Division of Health and Senior Services has access to all NHs in the state for the purpose of collecting satisfaction data. Resident and family satisfaction scores, along with cost and other quality indicators, will be used to reward the best facilities with a "Gold Seal" (Title XXIX, Ch. 400.235). Florida produces a written nursing home guide and "report card" system (also available on the World Wide Web) that includes facility demographics, contact information, and results of the annual on-site licensure and Medicare–Medicaid certification inspection.
Survey Development
Florida developed a "Client Satisfaction with Long-Term Care" instrument and established a goal of conducting face-to-face interviews with approximately 30 residents per facility. For smaller facilities, 70% of residents were to be included (up to 30 residents). For larger facilities, 10 of the sampled residents were to be short stay (rehabilitation–subacute), and 20 residents were to be long stay (NH). A five-question cognitive screen was developed for long-stay residents to determine competency to participate; it was derived from the MMSE and administered by facility staff.
An expert panel developed the instrument and cognitive screen by combining items from published and industry-supplied survey instruments. This produced instruments of 14 items (for long-stay residents), 7 items (for short-stay residents), and 12 items (for families). The core content of the two resident instruments is identical, but those items relating only to long-stay residents were excluded from the short-stay questionnaire. Similarly, the family instrument was adapted from the long-stay instrument, keeping the content of the items identical but with minor word changes and including additional items (e.g., "Does staff tend to keep the resident comfortable?"). Responses for all three instruments are recorded on a 5-point Likert scale. The psychometric properties of the instruments were not evaluated.
Administration and Cost
The resident instrument is to be administered face-to-face by an outside contractor. Similarly, the family instrument is to be mailed by an outside contractor, who will then collect and analyze the data and provide a written report to the Division. The Division is currently administering the family instrument "in house" and mailing instruments to each facility for distribution (the Division reported a 54% response rate for this family questionnaire). Legislation and regulations have established the goal of performing 22,000 face-to-face interviews with residents and mailing out 75,000 satisfaction instruments to families annually, with funding shared between Medicaid (30%) and federally unmatched state funds (70%). The estimated per facility cost was $732 (S. Phelps, personal communication, Florida Managed Care and Health Quality Agency for Health Care Administration, September 17, 2001). In 2001–2002, when many states experienced budget difficulties, the implementation of this system in Florida was postponed, illustrating the financial challenges faced by states seeking to find resources for new quality initiatives.
Iowa
Legislation and Design
Iowa requires the assessment of resident satisfaction as part of its annual ALF licensure and certification procedures but not for nursing facilities. State personnel reported using three survey instruments for assessment of resident satisfaction in ALFs: a resident instrument, a family instrument (modified from the resident instrument), and a separate staff instrument. The satisfaction instruments were developed by an outside contractor experienced with ALF satisfaction measurement and tailored to Iowa's requirements. Iowa provides the names and addresses of residents to the contractor, who then mails the instrument with a cover letter to all residents and has the completed instruments mailed back in a self-addressed, stamped envelope. The contractor also provides data analysis and report preparation.
Administration and Cost
The state program began in 1997 and eventually will include all ALFs. Satisfaction measures are administered at all facilities shortly before their annual on-site licensure and certification inspections, which translates into resident satisfaction assessment at 1- to 2-year intervals. The time period between facility inspections is determined by the scope and severity of the deficiencies received during the previous inspection, complaints substantiated, and the report of the inspectors' on-site visit. The Division of Elder Affairs inspectors (called "Recertification Monitors") share the information garnered from the satisfaction measurement process with individual facilities during the on-site inspection. The monitors also meet with small groups of residents, drawn at random from the facility's census at the time of the on-site visit, to validate the content and representativeness of the results obtained from the mailed instrument. The facility administrator is given a copy of the mailed instrument results and follow-up interviews, and he or she may or may not make them available to residents and families.
Iowa's program is funded through licensing fees, a research grant, and general revenue. The estimated per survey cost is $12 (A. Martin, personal communication, Iowa Division of Elder Affairs, June 13, 2001). In 2002, Iowa's program was temporarily suspended, and Iowa is investigating some logistical problems with the data collection and reporting system.
Ohio
Legislation
In 2000, the Ohio legislature passed a bill (123 G.A., H.B. 403) creating a "Consumer Guide" for Ohio residents in written and electronic form, which would provide information on long-term care options and quality information. Information on clinical quality indicators will be updated quarterly. The results of the annual on-site licensure and Medicare–Medicaid certification inspection, as well as resident satisfaction information, will be updated annually. The legislation set the goal of surveying satisfaction in 100% of facilities, using a mailed instrument for families and face-to-face interviews with residents. Cost is shared by facilities (charged $400 per year for this service), and by the state (R. Hornbostel, personal communication, Ohio Department of Aging, July 16, 2001).
Survey Development
Two research centers specializing in geriatrics were contracted to develop and test Ohio's NH resident and family satisfaction instruments. A four-question cognitive screen adapted from previous clinical research (Ejaz et al., 1994) was developed and tested. Stakeholder input, piloting, and psychometric evaluation were done under the deadlines mandated for statewide implementation. Family instruments were sent to facilities to distribute and return to the research centers.
Administration and Cost
In 2001, Ohio solicited contractors to administer the resident instrument, allowing for a variety of data collection options (e.g., facility volunteers, Ombudsman Program volunteers, and professional interviewers). Resident instruments are currently administered face-to-face by trained professional interviewers from a contractor experienced in geriatric and NH research. Estimated per resident cost is $25 (J. Straker, personal communication, Scripps Gerontology Center, September 20, 2001).
Oregon
Survey Development
The Oregon Division of Elder Affairs developed a satisfaction instrument for all elders receiving state-regulated residential health care services. Division personnel reported that items for the measurement instrument were adapted from "industry standard instruments." An initial 35-item face-to-face pilot survey was conducted by Division personnel and targeted 30 residents in each type of eldercare setting (NH, ALF, adult foster care, respite care, and home care services). Through statistical analysis, the number of items was pared to 9, with the use of the criterion that each item be applicable across all care settings. Oregon uses the same instrument regardless of long-term care setting type, which is coded for a particular living situation.
Administration
Oregon developed a method of resident screening called the Client Assessment Planning System (CAPS), completed annually by case managers for each client seeking residential services. This includes a physical assessment (activities of daily living) and a behavioral assessment (cognition, attitude or receptiveness to care, and safety). Residents receiving a behavioral assessment of "poor" are not included in the survey sample.
The satisfaction instrument is mailed annually with a return envelope to a randomly selected sample of residents from each type of elder care setting. No cost estimates were available (P. Jensen, personal communication, Oregon Department of Human Services, July 13, 2001).
Texas
Survey Development
Texas began measuring resident satisfaction in NHs in 1999. Three expert panels provided input into the creation of the Nursing Facility Performance Monitoring Data Instrument, which contains 13 items. Items were selected from the American Health Care Association nursing home satisfaction instrument. The panel added items for urinary management, physical activities, medication management, and meals (meals were the most common complaint received by the Texas Health and Human Services Complaint Bureau). The instrument was validated and interrater reliability evaluated by researchers from the University of Texas at Austin and the Texas Nurses Foundation.
Administration
Visiting nurses enter an NH and use the MDS to generate a list of residents from which to select a random sample, conduct a cognitive screen using the Short Portable Mental Status Questionnaire (Pfeiffer, 1975), and administer the instrument. If residents are unable or refuse to respond, the family is contacted by telephone and the same instrument is used to secure a proxy satisfaction score from a family member. The goal is to survey all NHs (1,250) annually by using a minimum of five residents per facility. Division personnel reported that "Texas is not interested in developing a ranking system for facilities for this Rider 26 project, but instead is striving to implement varied quality improvement strategies to benefit nursing facility residents" (L. Reyes-Gates, personal communication, Texas Department of Human Services, December 6, 2001). Funding is provided by the legislature with matching funds from the CMS. Cost estimates for the statewide program were not available.
Vermont
In 1999, the Vermont Department of Aging and Disabilities contracted with a private firm to measure the satisfaction of individuals receiving Vermont long-term care services (adult day care, Medicaid Waiver services, homemaker services, and attendant services). Satisfaction items as well as quality of life items were included in the instrument. A statistical sampling plan was used to form a representative sample drawn from 11 counties. A sample of individuals from the Vermont general population was also drawn as a comparison group for the quality of life questions. Data collection was carried out by the contractor using a combination of mailed and telephone interviews. Under legislation passed in 1998 (33 V.S.A. 
7112 Sec. 3), the program will be conducted annually and will eventually include the development of resident satisfaction measures in nursing homes and a quality reporting system on the World Wide Web. Cost estimates were not available (J. Senecal, personal communication, Vermont Department of Aging and Disabilities, July 23, 2002).
Wisconsin
Wisconsin passed legislation, that is, HFS 83.32 (2) 8(b), mandating the use of satisfaction measures in their licensed Adult Day Care centers in 1996. The state developed its own resident satisfaction instrument and offers facilities the option of using the state-developed instrument or designing or purchasing their own. Facilities must administer the instrument annually and state surveyors review the results at the time of recertification. Facilities also are allowed to determine the method of survey administration, including inclusion and exclusion criteria, and to summarize the results. No cost estimates were available (J. Eakins, personal communication, Wisconsin Department of Health and Family Services, July 19, 2001).
|
Discussion |
|---|
|
TOPAbstractMethodsResultsDiscussionReferences |
|---|
Several measurement and implementation problems for consumer satisfaction initiatives that were identified in the research literature were also identified from our survey of state satisfaction measurement programs. The first measurement problem is instrument validity. Although a consumer-driven approach would suggest that the content of satisfaction instruments should be directed by what is important to residents themselves, rather than from an exclusively professional perspective, only New Jersey and Ohio reported the use of qualitative research with residents to determine the content of their respective NH instruments. Without such input, measures may not address residents' values and perceptions and may lead, as has been pointed out in the satisfaction literature, to "ceiling effects," making statistical comparisons within and across facilities difficult if not impossible to detect. Additionally, instruments developed for ALF populations may not be valid for NH populations (and vice versa), as well as instruments developed for measuring family satisfaction and not resident satisfaction (and vice versa).
Another key tool in establishing validity is instrument psychometric testing, without which the reliability and validity of results is quite uncertain. Only Iowa, New Jersey, Ohio, and Texas report testing their instruments for psychometric properties and with their target data collection population. States that do not utilize psychometrically evaluated instruments may be able to use satisfaction data for feedback to individual homes for quality improvement efforts, but such results are of questionable utility for comparative performance evaluation.
A second measurement problem is the number of items to include in an instrument. The number of items included by each state in their satisfaction instrument varied from 7 (Washington) to 70 (Iowa). This variation reflects the two general uses of satisfaction measurement results: benchmarking and quality improvement. Using results for benchmarking may require fewer items, focusing on a more global perspective, whereas using results for detailed feedback and to stimulate facility CQI programs may require multiple items or subscales. This choice should be considered early in the instrument development process. It should be noted that states also varied in their instrument administration technique (mailed, face-to-face, or telephone) and the populations they were measuring (NHs, ALFs, or both). A short instrument may be warranted when one takes into consideration the issues of cost, respondent burden, and the declining ability of elderly residents to answer questions meaningfully over a lengthy interview period (Zinn, Lavizzo-Mourey, & Taylor, 1993). For example, preliminary pilot studies in Oregon and a state-funded research project in New Jersey found that an instrument of approximately 35 items took interviewers from 10 to 140 min (M = 30) to complete when face-to-face with residents. Both states used the results of piloting with a longer instrument to determine the best items for a short-form instrument. Oregon's final instrument contains 9 questions. New Jersey's is projected to contain approximately 20, reflecting an effort to balance inclusion of diverse aspects of care quality and the ability to discriminate between individual facilities.
A third measurement problem is the response rate obtained by the administration technique. Instruments mailed to elder populations can be less costly to administer than face-to-face interviews, but the trade-off may be lower response rates (Crawford, 2000). For example, aggregate data from Florida (where the response rate was reported to be 54%) and Oregon (where it was reported to be 51%) indicate that residents and families from some facilities may be underrepresented, which could bias comparisons between facilities. This issue is further complicated by the potential for wide fluctuation in response rates among different facilities and facility types, making statistical analysis difficult at best.
A fourth measurement problem is the use of cognitive screens. Despite substantial clinical evidence recommending cognitive evaluation for long-term care respondents, only Florida, New Jersey, Ohio, and Texas use a separate cognitive screen at the time of resident satisfaction measurement. Florida has not tested its modified version of the MMSE for validity and reliability. New Jersey assessed the validity of the MDS-COGS as a screening tool for cognitive impairment by comparing it with results of the MMSE given at the time of satisfaction measurement. Researchers found a moderately high correlation (r = –.47) between the two instruments and are in the process of determining a cognitive threshold for resident inclusion in satisfaction measurement. Ohio has utilized five items from previous clinical research (Ejaz et al., 1984), but with mixed results for determining levels of cognitive impairment. Similarly, Texas' instrument (Short Portable Mental Status Questionnaire) has been found to have lower sensitivity in distinguishing between levels of cognitive impairment in comparison with the MMSE (Fillenbaum, 1980; Smyer, Hofland, & Jonas, 1979).
Including cognitively impaired residents is especially important in NHs, where it is well known that 40–60% of residents have some level of cognitive impairment. Although several past research studies have eliminated even the minimally cognitive impaired or used proxies in situations where residents were not found competent to give informed consent, Simmons and associates (1997) have demonstrated that even severely cognitively impaired residents may be included in quality measurement efforts when standardized screening criteria are used. Further, family, friends, or staff responses represent a different perspective from that of the resident and are not direct proxies for the resident's experience (Berlowitz et al., 1995; Gesell, 2001; Lavizzo-Mourey et al., 1992). Residents and families have different concerns in relation to quality of care, and they vary in their ratings of salient features of a long-term care facility (Levin, 2001). Their perspectives reflect different values, expectations, and day-to-day experiences with care (Ejaz, 2000; Levin, 2001; M. A. Smith, 2000). Such findings suggest that rather than the consideration of family members' evaluations as proxies for those residents when the latter cannot provide valid responses, their responses are probably best viewed as distinct information and analyzed separately.
A fifth measurement problem is resident selection bias. As has been previously discussed, the use of cognitive screens may allow for the inclusion of cognitively impaired individuals to the extent that they are able to respond validly and completely to the items. Iowa mails questionnaires to an ALF's total resident population shortly before conducting their annual licensing and certification inspection. Ohio attempts to include all NH residents that have passed their selection protocol, including a cognitive screen. Florida, New Jersey, Oregon, and Texas utilize simple random samples of residents drawn from the facility census. Florida and Texas administer a standardized cognitive screen to residents selected for satisfaction measurement at the time of interview; New Jersey prescreens residents by using MDS data (MDS-COGS). As noted previously, Oregon screens residents by using their CAPS. Each method presents opportunities for the introduction of selection bias through methods that broadly exclude cognitively impaired individuals, use subjective or nonstandardized screening criteria, or lack adequate training of the persons selecting and administering the items to residents. We suggest the use of sampling methods and standardized screens to avoid selection bias. We further suggest caution in selecting contractors that may lack geriatric or long-term care research skills and may use inadequately trained interviewers. Several examples illustrating possible sources of bias that may limit the usefulness of satisfaction data follow.
Oregon's CAPS, which relies on case manager evaluations of residents' behavioral status, may be subjective. Although an evaluation instrument is provided to case managers, Oregon defines a resident's behavioral status in terms of any behavior that "places demands on others." This broad criterion not only guarantees the exclusion of the severely cognitively impaired for satisfaction measurement but might also exclude mildly to moderately impaired residents as well. In addition, many case managers are not clinical professionals (e.g., nurses or social workers), possibly adding another layer of subjectivity. To Oregon's credit, a comprehensive training program for case managers has been developed, thus offering an opportunity to reduce potential bias and to increase responsiveness to resident needs (F. Pfohman, personal communication, Oregon Department of Human Services, July 14, 2001).
Iowa, Oregon, and Wisconsin's techniques do little to avoid the possibility of the biasing of satisfaction results by facility staff and management toward more positive ratings, deliberately or otherwise. Wisconsin and Iowa rely on facility staff's judgment to determine who should be included in the measurement process, a procedure that could leave the door open to disproportionate inclusion of residents expected to give positive results. Where the facility staff selects subjects or administers instruments, residents might also be put in the position of feeling undue pressure to rate the facility positively. Iowa has recognized this problem and is attempting to increase the reliability of measurement results by using follow-up interviews with residents at the time of the facility on-site licensure and certification inspection. Further methodological problems in these states include the nonrandomized sample selection process and, in the case of Iowa and Oregon, the potential for mixing resident and family responses, because it is often difficult if not impossible to determine who filled out the mailed instrument. Identical instruments and procedures that mix resident and family responses compromise efforts to measure what have been shown in research studies to be distinct perspectives.
Two state implementation problems were also identified. The first is the potential for conflict between state satisfaction measurement efforts and the measurement efforts of provider organizations or individual facilities. In some states where satisfaction measures have not been initiated, provider groups have developed satisfaction measurement programs for consumer information (e.g., Michigan). Many individual facilities and corporations also have developed satisfaction measures for purposes of accreditation (e.g., Joint Commission on Accreditation of Healthcare Organizations) or marketing. In states seeking to use satisfaction measures for benchmarking facilities (e.g., Florida and New Jersey), these competing purposes may be at odds. States must evaluate carefully not only the potentially biased nature of provider-developed data collection efforts but also the potential burden on residents and families when both state and individual facilities seek to measure consumer satisfaction simultaneously.
The second implementation problem is the limited dissemination of satisfaction measurement results. Although all seven states pursuing annual satisfaction measurement identified quality improvement as their focus, only Florida, Iowa, and Wisconsin utilize the data in their annual on-site licensure and Medicare–Medicaid certification inspections. Additionally, only Florida, Ohio, and Texas provide a means for consumers to compare facilities based on quality or satisfaction information in written form or on the World Wide Web. Such limited use of satisfaction data by states serves to underscore the discrepancy between the opportunity for quality improvement, consumer information, and regulatory enforcement activities and the actual utilization of satisfaction data by the long-term care industry.
Resident and family satisfaction data could become part of a systematic, comprehensive information system, from which individual NHs and ALFs could routinely receive feedback from consumers and the state. For regulators and policymakers, such information could help bridge the gap between clinical outcomes information and consumer concerns. For consumers, satisfaction information could provide a more intuitive means for interpreting and appreciating the complex and often misinterpreted clinical quality indicators and deficiencies reported by some states from MDS and annual on-site licensure and Medicare–Medicaid certification results (Koenig & Kleinsorge, 1994).
Limitations of the Study
Although this first national study of state consumer satisfaction initiatives in NHs and ALFs achieved a 100% response rate, the limited number of states engaged in satisfaction measurement limits the scope of the descriptive findings. The survey did provide an opportunity for in-depth investigation of initiatives in progress in seven states, all of which have less than 3 years of experience in conducting satisfaction studies. For this reason, this survey of state initiatives should be considered as a preliminary step; as other states begin the process of satisfaction measurement, other implementation, administration, and cost issues, as well as best model approaches, may surface.
Conclusion
On the basis of evidence from our study, we conclude that satisfaction measurement by states is in an early stage of development. Currently, there is a general lack of rigor in instrument development (i.e., few instruments have been evaluated for their psychometric properties), and there is a lack of consensus on who are the consumers, how often to measure, what criteria should be used to select residents for satisfaction studies, and how to guard against bias. Additionally, current state measurement programs primarily seek to measure resident satisfaction with less attention paid to the measurement of family satisfaction, which is sometimes viewed as a proxy for resident perceptions rather than as a reflection of a separate and distinct perspective. A significant deterrent to development and maintenance of state initiatives may be the relatively high cost and the complexity of measuring resident and family satisfaction. Long-term care satisfaction data currently collected by states are primarily used for facility benchmarking or for consumer information, with little attention paid, as yet, to their potential for supporting quality improvement programs.
Our findings highlight the need for using standardized data collection instruments and data collection techniques for comparisons of facilities both within and across states. Further development is also needed to improve the dissemination of satisfaction information (and meaningful quality information in general) so that facilities, residents, families, and state administrators and regulators have the necessary tools to select, monitor, and improve the quality of long-term care systems. These processes have the potential to stimulate a greater cooperative effort on the part of all stakeholders to improve not only the quality of resident care but also the quality of resident life.
|
Footnotes |
|---|
1 School of Social Work, Rutgers University, New Brunswick, NJ.
2 Rutgers University, Institute for Health, Health Care Policy, and Aging Research, Rutgers University, New Brunswick, NJ.
4 College of Nursing, Rutgers University, Newark, NJ.
Decision Editor: Laurence G. Branch, PhD
Received for publication March 25, 2002. Accepted for publication September 25, 2002.
|
References |
|---|
|
TOPAbstractMethodsResultsDiscussionReferences |
|---|
This article has been cited by other articles:
 |
|
 |
|
  E. A. Miller and V. Mor Balancing Regulatory Controls and Incentives: Toward Smarter and More Transparent Oversight in Long-Term Care Journal of Health Politics Policy and Law, January 1, 2008; 33(2): 249 - 279. [Abstract] [PDF]
|
|
|
|
 |
|
 E. Sikorska-Simmons Linking resident satisfaction to staff perceptions of the work environment in assisted living: a multilevel analysis. Gerontologist, October 1, 2006; 46(5): 590 - 598. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
P. Edelman, M. Guihan, F. B. Bryant, and D. J. Munroe Measuring resident and family member determinants of satisfaction with assisted living. Gerontologist, October 1, 2006; 46(5): 599 - 608. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
G. Arling, R. L. Kane, T. Lewis, and C. Mueller Future Development of Nursing Home Quality Indicators Gerontologist, April 1, 2005; 45(2): 147 - 156. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
N. G. Castle and T. J. Lowe Report Cards and Nursing Homes Gerontologist, February 1, 2005; 45(1): 48 - 67. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 N. G. Castle Resident Satisfaction in a Transitional Care Unit Journal of Applied Gerontology, December 1, 2004; 23(4): 411 - 428. [Abstract] [PDF]
|
|
|
|
 |
|
 N. Castle Family satisfaction with nursing facility care Int. J. Qual. Health Care, December 1, 2004; 16(6): 483 - 489. [Abstract] [Full Text] [PDF]
|
|
|
|
|
|
N. G. Castle, T. J. Lowe, J. A. Lucas, J. P. Robinson, and S. Crystal Use of Resident Satisfaction Surveys in New Jersey Nursing Homes and Assisted Living Facilities Journal of Applied Gerontology, June 1, 2004; 23(2): 156 - 171. [Abstract] [PDF]
|
|
| ||||||||||||||||||||||||||||||||
| HOME | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|---|
