| HOME | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|---|
| ||||||||||||||||||||||||
 |
Abstract |
|---|
 TOP Abstract MethodsResultsDiscussionReferences |
|---|
Key Words: Indicators • Quality information • Choice • Performance • Health care consumers • Regulated competition
Current Approaches and Their Problems
The key question when developing a quality report card is what information it should contain (i.e., what the appropriate "building blocks" are). In this article we use building blocks, quality themes, and quality domains as synonyms. Building blocks (such as "ambience and privacy in a nursing home") can be subdivided into quality aspects (e.g., "feeling at home"), which in turn can be measured by quality indicators ("percentage of clients who say they feel at home when they are asked"). The literature describes several approaches to developing quality report cards (e.g., see Harrington, O'Meara, Kitchener, Payne Simon, & Schnelle, 2003; Mattke, Reilly, Martinez-Vidal, McLean, & Gifford, 2003). In the United States, for example, preliminary versions of the Centers for Medicare and Medicaid Services (CMS) Nursing Home Compare Web site (the largest and most well-defined nursing home report card in the United States) were mainly based on the work of researchers at the University of Wisconsin's Center for Health Systems Research and Analysis. These researchers developed a set of nursing home quality indicators from existing items in the Minimum Data Set (MDS), a set of 15 clinical outcome indicators such as pressure ulcers and weight loss (K. Berg et al., 2002; CMS, 2005; Zimmerman et al., 1995). Another way of developing a report card is to look at existing cards (in other countries; Castle & Lowe, 2005) or to ask the public what it wants to know about health care quality (Hibbard & Jewett, 1996). Other techniques include Delphi techniques (Normand, McNeil, Peterson & Palmer, 1998), storytelling methodology (Sofaer, Gruman, Connaughton, Grier & Maule, 2000), and the Balanced Scorecard approach (Hall et al., 2003).
All of these methods share the same two problems, however. First there is the trichotomy between the quality report card's technical validity (do the indicators measure what they are supposed to measure?), its appropriateness (does it contain the information consumers need and want?) and its feasibility (are measurable data available?). Selecting indicators from the literature, from carefully chosen existing quality report cards, or on the basis of expert opinion may guarantee validity and technical adequacy but will not necessarily fit with consumers' perspectives. However, giving consumers carte blanche when building a quality report card might lead to a rather utopian set of indicators that is hard to measure or that goes beyond common and feasible health policy objectives, and as such constitutes an undesirable incentive mechanism for care providers from the health care perspective.
M. Berg and colleagues (2005) stressed the importance of feasibility of public performance data. They argued that feasibility is the most important prerequisite for public reporting, more than perfect technical validity of the indicators, say. The appropriateness dilemma is well illustrated in studies by Arling, Kane, Lewis, and Mueller (2005), Castle and Lowe (2005), and Harrington and colleagues (2003). Arling and colleagues critically evaluated nursing home quality indicators used in the United States for measuring and comparing nursing homes, including the CMS Nursing Home Compare indicators. One of their conclusions was that quality indicator reporting systems should be tailored to specific stakeholder needs, for example to consumers. The second study examined nursing home report cards in 19 U.S. states and found that the information presented differed substantially. Castle and Lowe concluded that this variation in types of information—caused by the specific needs of regulators, provider organizations, and consumer groups—imposes a considerable limitation on current quality report cards, as it creates a situation in which few elderly people are capable of judging report card information. Based on a literature review, Harrington and colleagues presented a rationale and a framework for presenting comprehensive consumer information on nursing homes through report cards. They found six key information areas: (a) facility characteristics and ownership; (b) resident characteristics; (c) staffing indicators; (d) clinical quality indicators; (e) deficiencies, complaints, and enforcement actions; and (f) financial indicators. However, their claim that all of these areas are important in informing consumers fully was not substantiated by testing their findings against consumer preferences.
The second problem with current approaches is the dichotomy between the "ivory tower strategy" and the "consensus strategy." Some report cards are developed by one or a very limited number of parties. These report cards often lack consensus and support and are consequently rarely fully implemented or have a short lifespan. Castle and Lowe (2005) warned that they had based their cross-section of 19 report cards on the opinion of only one research team and that their results should be presented to consumers before any conclusions could be drawn regarding their appropriateness. Other report cards are developed jointly by regulators, providers, insurers, and consumers. Such a consensus process usually leads to a greater diversity of information, making it much more difficult for users to understand the information provided, make trade-offs between information items, use the information to prioritize provider organizations, and come to a well-considered decision (Jewett & Hibbard, 1996). In the United States, this problem is well illustrated by the CMS Nursing Home Compare Web site, probably the best known example of a consensus-seeking report card development process. From 2002 to 2004, the National Quality Forum developed 16 performance measures that
facilitate standardized comparison of the quality of nursing homes. These quality measures were carefully reviewed and endorsed by a diverse group of stakeholders: consumer and patient groups, health care purchasers, health care providers and health plans, research and quality improvement organizations. (National Quality Forum, 2004, p. E1)
However, despite the thoroughness of the consensus process, evaluations of the Nursing Home Compare initiative reported several problems, including the appropriateness of the indicators for consumers, as well as failure to identify key conceptual dimensions and to aggregate indicators into general categories or domains, which would make it easier for consumers to understand information (Arling et al., 2005; U.S. General Accounting Office, 2002).
What is needed, apparently, is a method for developing quality report cards that strikes a balance between validity, appropriateness, and feasibility of content while generating sufficient consensus and support in the development process. In this article we present an approach to meet this challenge that was recently developed and applied in The Netherlands.
Objectives
This article (a) reports on a study conducted to identify the appropriate building blocks for quality report cards for geriatric care from the consumer's perspective and (b) aims to describe the results of the study—the building blocks—plus the innovative step-by-step approach developed to arrive at these results. The article ends with a discussion of the results and methodology, plus some suggestions for future quality report card initiatives and applications of the method.
|
Methods |
|---|
|
TOPAbstractMethodsResultsDiscussionReferences |
|---|
William Trochim first introduced Concept Mapping in 1989 as "a type of structured conceptualization which can be used by groups to develop a conceptual framework which can guide evaluation or planning" (Trochim, 1989). Over the past 15 years, experts have used Concept Mapping in areas other than evaluation and planning (e.g., for defining and assessing quality of care; Trochim & Kane, 2005). Concept Mapping can take various forms, such as Idea Mapping, Mind Mapping, Causal Mapping, or Cognitive Mapping (Trochim & Kane, 2005). Here we focus on the form used in the Trochim and Kane study, Structured Conceptualization. This is a mixed-method, participatory, group idea-mapping methodology that integrates well-known group processes such as brainstorming (Bowling, 2002) and unstructured sorting (Coxon, 1999; Weller & Romney, 1988) with a sequence of multivariate statistical methods. From this point forward, we use the term Concept Mapping.
In its most extensive form, Concept Mapping comprises six steps (Trochim & Kane, 2005): (a) In the preparation step, the focus for the mapping project is identified, participants are selected, and the project schedule and logistics are determined; (b) the generation of ideas and statements often involves some form of brainstorming, and the ideas generated are synthesized into statements; (c) participants then sort the statements and rate them according to one or more variables (e.g., importance, feasibility); (d) multivariate statistical analyses of multidimensional scaling (Davison, 1992; Kruskal & Wish, 1978) and hierarchical cluster analysis (Anderberg, 1973; Everitt, Landau, & Leese, 2001) are used for the representation of the ideas in maps; (e) next, participants are involved in the interpretation of these maps; (f) finally, in the utilization phase, the maps and all associated information are used to define the concepts (e.g., quality themes or purposes of a project).
Table 1 summarizes the Concept Mapping procedure as used in this study (i.e., identifying the appropriate building blocks for quality report cards for geriatric care from the consumer's perspective).
|
|
Results |
|---|
|
TOPAbstractMethodsResultsDiscussionReferences |
|---|
Ideas and Statements
The Concept Mapping process resulted in two lists of quality aspects, criteria, and indicators, some 350 for home care and 400 for institutional care. We condensed these into a short list of 88 statements for quality of home care and 90 statements for quality of institutional care. Tables 2 and 3 present a selection of these statements.
|
|
Cluster Analysis
Initial analysis of the clustering of statements by experts revealed 17 clusters for quality of home care and 18 for quality of institutional care. In both cases, we reduced the number of clusters step by step using Concept Mapping software (www.conceptsystems.com). We analyzed each step, taking into account the clusters being grouped together in that step and the content represented by the newly formed clusters. This resulted in 10 clusters for quality of home care (one of which, "Remainder," contained a noncoherent set of statements) and 8 clusters for quality of institutional care as potential building blocks for future quality report cards.
Cluster Interpretation
Next, we presented the clusters to the expert panels, who discussed them with respect to content validity, clarity for consumers, and appropriateness of the proposed labeling. The home care expert panel proposed spreading the statements of one home care cluster ("Remainder," Cluster 3) over the other clusters. The institutional care expert panel suggested splitting two institutional care clusters (5 and 7) and distributing half of the statements in Cluster 5 over the other clusters. This left nine clusters (or quality themes) for both home care and institutional care for the future quality report card. The draft final maps, clusters, and aspects were also discussed during focus groups with consumers. No major amendments were required, merely a few changes in wording and labeling to increase clarity for consumers. Tables 2 and 3 present the final clusters, ranked according to their mean rating by consumers. The tables also show a selection of the five statements that consumers rated highest, the experts' mean statement and cluster ratings, and the bridging scores.
|
Discussion |
|---|
|
TOPAbstractMethodsResultsDiscussionReferences |
|---|
Reflection on the Results
This study showed that, although home care and institutional care for elderly adults share many quality themes, the two types of geriatric care need separate quality report cards.
Differences and Similarities Between Home Care and Institutional Care
Early in the study, we decided to develop two separate quality report cards for geriatric care: one for home care and one for institutional care. This decision was based on the information gathered during the expert interviews and was reaffirmed later by the differences in content and consumer ratings between the quality themes. "Availability, continuity, reliability, and organization of care" (Cluster 1), "waiting time" (2), "complaints" (6), and "informal care" (8) were defined for home care and not for institutional care (see Table 2). "Protocols and procedures" (Cluster 5), "ambience and privacy" (6), "quality of rooms" (8), and "organization of care" (9) were unique to institutional care (see Table 3). We found that consumers of home care and institutional care attached different values to corresponding quality themes. For instance, institutional care consumers rated the themes "privacy, respect, and autonomy" and "participation and choice" higher than home care consumers, whereas the latter found the themes "availability, continuity, reliability, and organization of care" and "waiting time" more important. These differences may reflect the differing impacts of having to leave home for a nursing or residential home as compared to having caregivers come into the home setting. People moving to a new environment because of their need for care may be more preoccupied with preserving some autonomy, retaining some privacy, and having some say in their day-to-day schedule of care provision, (social) activities, meals, and so on. The high rating of "privacy, respect, and autonomy" supported this. People receiving care at home are more concerned with availability and reliability of caregivers, caregivers keeping appointments, having a limited number of different caregivers, and organizing care provision in such a way that it fits in with their home and family life.
There was a striking similarity in the high ratings attached to "staff expertise" and "personal care plan and care file." The underlying reasons appear to differ, however. In home care, the staff experience relates to quality of care in terms of safety and effectiveness, independence, and responsiveness to individual consumer demands. The personal care plan, by contrast, is important primarily because it constitutes the person's care entitlements, whereas the care file is important as a means of communication and coordination between the different caregivers. In the institutional care context, staff experience relates not just to effectiveness and safety of care, but also to the patient–caregiver relationship and caregivers' efforts to promote patient well-being. Quality of care appears to be seen more as a responsibility of the institution than of individual caregivers, and this is also reflected in the themes "protocols and procedures" and "organization of care." Furthermore, both the personal care plan and the care file are mainly relevant as a means of communicating with and rendering account to the patient and his or her family.
Differences and Similarities Between Consumer and Expert Perspectives
The most striking difference between consumers and experts was the experts' expectations that home care consumers would value responsiveness to consumer demands (and would hence give high ratings to "participation and choice" and "privacy, respect, and autonomy"). In the experts' view, the timeliness and effectiveness of home care would be less important to consumers, because most of the past waiting list problems in home care have now been resolved and consumers were believed to be less involved with the medical (outcome) aspects of care. Apparently, the day-to-day experiences of many home care consumers are different, given the high value they attached to receiving high-quality care from highly qualified health care workers and at the appointed times.
In the field of institutional care, experts and consumers largely agreed on the most important quality aspects when choosing a provider. A striking similarity was the low rating of aspects of organization of care that are usually propagated as quality performance indicators, such as the availability of a quality system and employee satisfaction.
Conclusions Regarding the Results
The findings show that, although home care and institutional care share many quality themes, separate quality report cards are needed for the two types of geriatric care. Home care consumers attach more value to the availability, continuity, and reliability of care, whereas consumers of institutional care value privacy, respect, and autonomy most. This study also shows, unlike many other quality report card studies, that consumers want information on structure, process and outcome indicators, and rating outcome indicators such as effectiveness and safety of care both for geriatric home and for residential care.
Reflection on the Methodology
This study shows how Concept Mapping can be used to identify building blocks for quality report cards. Integration of existing quality information sources and viewpoints of experts in the field of geriatric care supports the validity and feasibility of the content of the quality cards, whereas integration of consumer preferences supports its appropriateness. Furthermore, participation by all stakeholders helped to build consensus about the building blocks, which may be expected to facilitate implementation.
Limitations and Suggestions
We need to mention some limitations of the current study, but first it is important to look at a key assumption made at the beginning of the study. Developing a quality report card for elderly people presumes that they want quality information and that they will use report cards. Although this study did not seek to verify this assumption, prior (and some current) research has shown that such an assumption does not always hold (Castle, 2003; Cheek & Ballantyne, 2001; Wackerbarth, 1999). Further research is needed in The Netherlands to explore the decision-making processes of older adults and to determine how helpful quality report cards would be.
The first limitation concerns the number of respondents rating the statements (and hence clusters): Between 11 and 17 respondents participated in the focus groups. This is not so much a problem for the definition of clusters (i.e., quality themes), but it does limit the ability to generalize the prioritization of quality themes to larger groups of consumers. However, a second method that was used to sort the themes: Q-methodology revealed support for the ranking of quality themes as presented in Tables 2 and 3. Q-methodology (Brown, 1980; Stephenson, 1953; Van Exel & De Graaf, 2005) was used as a supportive tool during the focus groups; we do not discuss the method and its results in detail here. Q-methodology provides a foundation for the systematic study of subjectivity, a person's viewpoint, opinion, beliefs, attitude, and so on. Typically, a Q-methodological study presents people with a sample of statements about some topic, called the Q-set (here, the quality themes). Respondents are asked to rank the statements from their individual point of view, according to some preference, using a quasi-normal distribution (here, desirability for inclusion in a report card). These individual rankings are then subjected to factor analysis, resulting in factors representing operational clusters of subjectivity (here, both studies showed very similar results, indicating the existence of two archetypal preference structures: one that emphasizes availability, reliability, and continuity of care and the expertise of caregivers; the other that finds these same quality themes important but attaches higher value to demand-oriented care. In home care, this primarily refers to patient involvement in care decisions, so that home care provision fits in with the home or family situation and the informal caregiver network. In institutional care, it refers more to privacy, flexibility of services, and treatment by professionals, so that individuals moving into an institution are able to retain some level of autonomy). In this way, Q-methodology can be very helpful in exploring tastes, preferences, sentiments, motives, and goals (here, knowledge of the archetypal preference structures helped to assess whether potentially diverging views on items to be included on quality report cards (a) were adequately represented in the results from the rating exercise and (b) provided a broader perspective on the need, development, and use of quality report cards).
A second limitation may be the focus on existing quality information: There might conceivably be quality aspects that are very important to consumers but are not yet measured in any existing instrument. The focus groups addressed this by starting with an open brainstorming session, but it did not reveal any new quality items. Ongoing changes in the market structure and consumer preferences do, however, mean that experts should evaluate quality report cards regularly.
Third, in this study experts performed the sorting of statements, whereas in order to limit cognitive load consumers were only asked to reflect on this sorting (88–90 statements). Though in theory consumers could have sorted the statements, we found extensive support for the experts' clustering during the focus groups.
Finally, the double-barreled statements and quality aspects in this study may be problematic. For example, we grouped the items "the caregiver works independently, efficiently, carefully, and hygienically" into one quality aspect. This statement might not have meant the same thing to all respondents. Future research needs to examine the (relative) weight consumers attach to each of these items.
We can make some suggestions for improvement. Concept Mapping is a consensus method. Here, consensus was created among experts from various parts of the health care field and among consumers, with the two groups reaching this consensus independently of each other. The consensus effect might have been even stronger if experts had discussed the results with consumers. A further improvement would be to ask respondents (i.e., the experts) not only to rate the statements, but also to give a feasibility score. When quality aspects and indicators have to be selected for the quality report card, both the rating and the feasibility score can function as selection criteria (Nabitz, van den Brink, & Jansen, 2005; Trochim, 2005<--CO?3-->).
Embedding Study Results
Compared to other studies and initiatives in relation to performance indicators for care (see the introductory paragraphs to this article), the quality themes identified in the current study are more about quality and less about facts (i.e., aspects of structures like facilities, residents, deficiencies), just as the Online Survey Certification and Reporting data partially fill the CMS Nursing Home Compare Web site in the United States (Cowles Research Group, 2006). This is obviously due in part to the aim of this study, which was to define building blocks for quality report cards. However, during the focus groups consumers said that quality information on report cards should be supplemented with factual information, an easy task using existing sources of information.
Nonetheless, comparing the quality aspects identified in other studies and initiatives such as Nursing Home Compare reveals important differences in how quality is defined. For example, Nursing Home Compare assesses quality by means of the MDS. In the current study, however, quality items that can be measured with clinical indicators (such as the aspects within Cluster 2 in Table 3) are only one side of the quality coin. Most of the quality themes for institutional care (Clusters 1, 4, 6, 7, and 8 in Table 3) and many quality themes for home care (Clusters 5, 6, 7, and 9 in Table 2) deal with quality of life. This is poorly measured by clinical indicators such as the MDS, which we cited earlier as an important deficiency of the Nursing Home Compare initiative (Arling et al., 2005; Kane et al., 2003).
Other studies ignore outcome measures completely and limit quality assessment to process or structure indicators. However, the current study shows that consumers are interested in a broad range of issues incorporating structure, process, and outcome aspects that measure quality (see Table 3). As discussed earlier, consumers assign the highest importance to outcome indicators. Unlike other studies, this study did not find consumers to be interested in cost and financial performance of care providers. This is probably related to the Dutch health care market structure, which has very small user fees and little danger of provider bankruptcy.
Because consumers were actively involved in the Concept Mapping approach adopted in this study, we expected that it would generate quality indicators that fit consumers' needs for information more closely.
Challenges for the Future
As stated, the aim of this study was not to develop the report card itself but to identify appropriate building blocks from the consumer's perspective. The Dutch National Institute for Public Health and the Environment will create the report card; this agency develops all consumer report cards in Dutch health care. The fact that one agency develops all report cards contributes to the homogeneity of health care report cards, making them easier for consumers to understand. However, some challenges remain before the report cards for geriatric care are ready for use.
First, the most important quality aspects per quality theme need to be selected. The rating of quality aspects by consumers seems to be a reasonable criterion; alternatively, the ratings by the experts could be used, or so could a mix of the two ratings. The number of aspects per theme could also be varied. For instance, themes with a higher average cluster rating could be assigned more aspects, or more homogeneous clusters (with a lower bridging score) fewer aspects.
Second, a system of checks and balances could increase the reliability of the report card, such that at least one indicator for each aspect is measured objectively (by recording of facts such as falls or complications) and one is measured through consumer consultation (e.g., patient experience questionnaires). The inventory of existing data sources in Step 2 of this study revealed the indicators that are already measured for each aspect and, more important, the indicators that are actually accessible and useful for filling the report cards. Four data sources were available: two consumer satisfaction instruments and two instruments of the Dutch Health Care Inspectorate. In the next section we discuss current developments in the health care system in The Netherlands and other Western countries that will ensure quality in long-term geriatric care and increase the availability, comparability, and reliability of data sources.
Third, the indicators and scores have to be translated into comprehensible, everyday language. Several authors have stressed the importance of the communicative aspects of report cards (Agency for Healthcare Research and Quality/CMS, 2006; Hibbard, Slovic, Peters, & Finucane, 2002). The key issue here is the layered construction of the report cards. Information should not be too detailed, but it needs to be detailed enough for those who want to learn more about how scores are composed.
Fourth, the Concept Mapping procedure should be repeated regularly to reflect changes in consumer preferences, the market structure, and provider performance. Once quality report cards are in use, health care providers are known to begin focusing on the quality aspects that are measured on the report cards in order to improve quality and (perhaps more important in their view) their league table rankings (Zinn, Spector, Hsieh, & Mukamel, 2005). This is a powerful and beneficial impulse, but focusing on a limited number of aspects over an extended period might lead to unintentional effects, such as short-sightedness (van Thiel & Leeuw, 2002). Although close correspondence between the core themes of providers' quality policy and current consumer preferences is clearly not a problem, it is worthwhile monitoring trends in quality on aspects that are currently less highly valued by consumers (but that may be of societal value).
Recent Developments
Recent developments in Dutch and other Western health care systems have created a number of opportunities and challenges for consumers faced with choices, and hence for the development of quality report cards. An important positive development is the creation of one uniform set of performance indicators for the whole Dutch geriatric care sector. This was recently done by a steering group comprising all stakeholders in Dutch geriatric care; the group was led by the Dutch Health Care Inspectorate and was assisted by us. A total of 40 indicators will be measured by health care providers themselves by completing MDS-based forms at the patient level (covering Cluster 3 in Table 2 and Cluster 2 in Table 3), through consumer consultation (using uniform questionnaires based on the Consumer Assessment of Health Care Survey methodology; covering Clusters 1, 2, and 4–9 in Table 2 and Clusters 1, 3, 4, and 6–9 in Table 3), and through surveillance by the Health Care Inspectorate (covering Cluster 5 in Table 3). Beginning in January 2007, all Dutch geriatric care providers must use these instruments to measure consumers' experiences every 2 years and must use the MDS indicators continuously or at least once a year. Providers must also report on their quality to the public, both in an annual report and through the national consumer report cards for geriatric care. All in all, this system of checks and balances seems to guarantee the feasibility of consumer report cards for geriatric care in The Netherlands. For the U.S. Nursing Home Compare Web site, a similar procedure, especially the use of consumer experiences (using Consumer Assessment of Health Care Survey questionnaires that had already been developed for nursing homes) would be beneficial to supplement the report card with currently absent quality-of-life data (Arling et al., 2005; Kane et al., 2003). A good example of what we propose here is Ohio's report card for nursing homes. This initiative combines both consumer-survey data with MDS quality indicators and gives consumers a broad overview of what they may expect from a nursing home (State of Ohio, 2006).
Some developments might not facilitate the creation of consumer report cards on geriatric care providers, but rather give cause for critical reflection. First, care for elderly people with a chronic condition is increasingly being redesigned into integrated care pathways. Consequently, people might become more interested in the quality of disease-specific care arrangements rather than the quality of specific care providers. Second, following the liberalization of the Dutch health care market, home care split into product groups, enabling the access of different types of providers. For instance, domestic help—the largest segment of home care—can be provided by a range of service organizations outside the health care sector. As a result, in the near future people may be more interested in the quality of providers with respect to individual product groups than in the quality of the total organization. Another consequence of the ongoing liberalization is the emergence of large conglomerates of care providers with local branches that may vary in type and quality of service provision. Such mergers reduce consumer choice and raise questions as to the level at which quality should be reported: organizational or local branch level (given that quality may already vary substantially between caregivers from the same provider, hence the consumer's focus on continuity and expertise).
|
Footnotes |
|---|
1 Institute for Health Policy and Management, Erasmus MC, Rotterdam, The Netherlands.
Decision Editor: Linda S. Noelker, PhD
Received for publication April 24, 2006. Accepted for publication August 31, 2006.
|
References |
|---|
|
TOPAbstractMethodsResultsDiscussionReferences |
|---|
| ||||||||||||||||||||||||
| HOME | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|---|
