| HOME | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|---|
| ||||||||||||||||||||||||
 |
Abstract |
|---|
 TOP Abstract MethodsResultsDiscussionReferences |
|---|
Key Words: Cognitive interviewing • Institutionalized adults • Nursing home residents • Questionnaires
Cognitive interviewing is a survey design method that is used in other settings to understand how respondents interpret and answer survey questions. Our purpose in this study is to demonstrate how qualitative cognitive interview (CI) methods may be used to refine questionnaires intended for NH residents. Our specific goals are to (a) ascertain that constructs being considered for a new preference assessment tool (PAT) are concordant with NH residents' understanding and interpretation; (b) simplify and clarify survey item wording to maximize comprehension; and (c) determine whether residents are able to match choices from the response scale to the survey items.
Background and Framework
The Minimum Data Set (MDS) is a standardized assessment tool intended to identify NH residents' health problems. It is required for all residents of Medicare- or Medicaid-certified NHs. Providers and consumers have raised a number of important questions about the MDS, including whether it adequately addresses NH residents' quality of life. In response, the Centers for Medicare & Medicaid Services has undertaken a significant revision of the MDS 2.0, referred to as the MDS 3.0, and is collaborating with the Department of Veterans Affairs to develop and evaluate a revised tool. A primary goal of this revision is to increase resident voice in the MDS assessment process.
Two expert panels were convened to help set the agenda for the revision. The panels, one of which was composed of NH providers and advocates, the other of long-term-care researchers, independently recommended that the Customary Routine (CR) section of MDS 2.0 be retained but changed from its current format. The MDS 2.0 CR section, which is intended to facilitate individualized care planning around resident preferences, consists of a checklist of 20 items relating to a resident's daily routines in the 12 months prior to admission. As such, MDS 2.0 CR items may capture what the resident had to do because of physical and environment constraints, rather than what the resident would have valued in a more supportive environment. In addition, the MDS 2.0 does not give NH staff approaches for ensuring that resident preferences are understood. Specifically, staff are not provided with explicit, validated questions or survey tools that have been shown to reliably elicit preferences from frail NH populations.
Members of the expert panels felt the existing approach failed to realize the potential benefits of the CR section, and they proposed that the MDS 3.0 CR section adopt a direct interview format. Because the MDS is administered by direct-care staff, both panels felt that satisfaction items relating to care should not be used. However, they suggested that any newly developed preference survey should build on domains targeted by a NH resident quality-of-life satisfaction survey developed and tested by Kane and colleagues (2003). Once it was clear that we would be significantly changing the CR section, one of the panels was also asked to rate the validity and feasibility of particular topic areas for eliciting resident preferences. In addition, that panel rated a response format of "very important," "somewhat important," and "not important" as highest priority for future testing.
Our interdisciplinary research team, which included clinicians, survey methodologists, psychometricians, and gerontologists, incorporated these suggestions in the development of the PAT. Our intent was to design the PAT so that it could be used as part of the admission MDS assessment (current CR approach), or as a stand-alone assessment at other points in time. Potential items and design for the PAT are grounded in Residential Care Quality, proposed as a primary domain of NH care quality (Saliba & Schnelle, 2002). The domain includes identifying resident preferences for day-to-day activities and incorporating care that encourages autonomy in choosing and completing activities. Once we selected the candidate items, we mapped them onto quality of life domains identified by Kane and colleagues (2003). We used this mapping approach because we hoped that the PAT items might assist NH staff in assessing preferences relevant to improving resident-reported satisfaction and quality of life.
Rationale for CI Testing Methods
Conventional pretesting methods evaluate nonresponse rates and interviewer feedback to identify problems with surveys (Forsyth & Lessler, 1991; Presser et al., 2004). However, assumptions that survey developers build into the items or response sets may not be shared or understood by the participants. If this disconnect exists, then the survey may be burdensome, and high response rates may mask significant measurement error that compromises the survey's validity (Collins, 2003; Willis, 2005). These concerns are particularly pertinent in surveys intended for older individuals, who may confront barriers to effective communication, including stereotypes of elders that ignore their individual differences, impaired communication skills associated with age-related sensory and cognitive changes, and generational differences in preferred communication styles (Caplan & Greene, 1999; Caplan, Haslett, & Burleson, 2005; Ryan & Butler, 1996).
A different pretesting method, cognitive interviewing, applies the principles of cognitive psychology to survey development. The respondents' task is conceptualized in four stages in the CI: interpretation, retrieval, judgment, and response selection (Tourangeau & Rasinski, 1988). In Stage 1, question interpretation, participants must translate the item into a relevant attitude structure. In Stage 2, respondents retrieve beliefs and feelings relating to the attitude structure. The third stage involves assessment and weighing of attitudes to arrive at a judgment. Finally, after the answers are edited for consistency and social desirability, the judgment is "mapped" onto the available response options. Each stage is subject to cognitive failures.
In cognitive interviewing, researchers use participants' narrative reports about their cognitive processes in order to identify sources of response error and refine items before proceeding with conventional pretesting (Forsyth & Lessler, 1991; Presser et al., 2004). CI studies typically report small sample sizes. While this is sometimes cited as a shortcoming, CI methodologists counter that the goal of CI techniques is not statistical estimation but rather the identification of sources of response error within the group to be surveyed (Caspar, Lessler, & Willis, 1999; Willis, 2005<--CO?1-->).
The two principal techniques for conducting CIs are referred to as "think-alouds" and "verbal probing" (Collins, 2003; Presser et al., 2004; Willis, 2005). Using the think-aloud approach, the interviewer asks respondents to explain what they are thinking as they answer questions. This open-ended approach may be difficult and burdensome for participants not used to identifying or verbalizing the cognitive processes required to answer questions. In the alternative CI technique, verbal probing, the interviewer employs prompts to simplify the respondents' task. The interviewer first asks participants to answer each pilot item and then uses follow-up questions to prompt them to explain their cognitive processes. Interviewers may use probes scripted in advance, or they may use spontaneous unscripted follow-up questioning. Researchers may also observe and code respondent behavior and response latency during the CI (Presser et al., 2004; Willis, 2005).
CI techniques have been used successfully with healthy community-dwelling seniors. However, the usefulness of CI methods is not well established in more frail populations, including those residing in NHs (Jobe & Mingay, 1990). A few researchers who address frail elders have mentioned the use of CI, but they do not elaborate on the extent, methods, or influence of the process on item selection.
In this article we describe the use of CI techniques to pretest items for the PAT in a sample of NH residents. Moreover, we provide examples to illustrate how CI findings informed the PAT. We also report the results of a content analysis that we conducted to quantify the categories of problems identified during the CI process.
|
Methods |
|---|
|
TOPAbstractMethodsResultsDiscussionReferences |
|---|
22. Respondents were told that participation was voluntary and they could terminate study interviews at any time. No incentives were offered.
Interviews
Two postdoctoral gerontology fellows conducted the interviews in a location where residents confirmed they felt comfortable talking about their preferences. The interviewers used dual independent coding to report residents' verbatim responses and record behavioral observations.
We tested candidate items in sets to decrease respondent burden. Each resident was interviewed between one and four times during the 14-week cycle of testing, with sessions ranging in length from 7 to 46 minutes, depending on the number of items in the set. We had more than one round of interviews conducted for each set of items; this iterative approach allowed item replacement or refinement between interviews.
Item Selection for CIs
On the basis of the expert panel recommendations and a comprehensive literature review of existing preference questionnaires suitable for use in the NH (e.g., Carpenter, Van Haitsma, Ruckdeschel, & Lawton, 2000; Kane & Degenholtz, 1997; Kane, Degenholtz & Kane, 1999; Polisher Research Institute, 2006), the research team identified 516 candidate items.
To narrow this list, the team met several times to assess candidate items based on the following criteria: relevance to facility interactions in the residential quality framework (Saliba & Schnelle, 2002); high topic validity and feasibility ratings from the expert panel, the likelihood of fit in a structured survey, and the frequency with which a topic appeared in quality-of-life and preference literature (Carpenter et al., 2000; Kane & Degenholtz, 1997; Kane et al., 1999, 2003; National Citizens' Coalition for Nursing Home Reform, 1984; Polisher Research Institute, 2006; Rantz et al., 1999; Van Haitsma, 2000; Whitlatch, Feinberg, & Tucke, 2005); and the inclusion of items representative of each quality-of-life domain (Kane et al., 2003). The team modified the wording of items to match the common root question ("While you are here in the nursing home, how important is it to you to...") and the 3-point importance scale (very important, somewhat important, not important). The team ultimately selected 30 items for pretesting from the list of 516 candidate items.
Procedure
In order to standardize the interviews and minimize respondent burden and interviewer effects (a frequent criticism of CI techniques), the research team developed a semistructured interview protocol. The interviewers asked respondents to rate each of the items by using the 3-point importance scale, which they showed to them as typed wording in a large font. Interviewers made use of default response options (refused–don't know, no response, or nonresponsive answer) to record incomplete or refusal answers. They also noted behaviors, comments, and latency when they asked the residents each question.
After each response, interviewers immediately asked the related scripted probes. The cornerstone probe asked residents to provide examples of key terms. For example, a question about resident preferences for using "tobacco products" was probed by asking, "Can you give me an example of the kinds of tobacco products you were thinking about when you answered the question?" Examples including cigarettes, cigars, pipes, and chewing tobacco were accepted as evidence that the general term was understood.
After providing examples, participants were asked additional prompts depending on their initial response. Participants who said a specific item was not important or only somewhat important were prompted to discuss whether it had ever been more important in the past. For example, participants who indicated that using tobacco products was not important or only somewhat important were asked, "Has using tobacco products ever been more important to you?" Those who said it was very important or who indicated that using tobacco products had been more important in the past were subsequently asked, "Are you able to use tobacco products as much as you would like here at the nursing home?" Interviewers were permitted to ask additional questions if further clarification was needed.
We compared residents' responses using the scale and their verbatim answers to the probes in order to identify discordance, an indicator of potential problems with an item or with the match between the item and response scale. We operationalized discordance as disagreement between a resident's importance rating and the narrative answer to the probes.
The research team met regularly to review problems identified during CIs. Problems identified during even one CI were reviewed by the team to consider whether alternative versions were needed (Rosal, Carbone, & Goins, 2003; Willis, 2005). If testing revealed problems with an item, we revisited the original 516-item list for replacements or possible revisions.
Analytic Approach
We classified the problems discovered during the CIs by applying coding guidelines and coding forms from the Question Appraisal System (QAS-99) manual (Willis & Lesser, 1999). The QAS-99 manual describes eight broad categories of characteristics that can influence response rates and measurement accuracy. They are as follows: reading (can questions be uniformly read?), instructions (are instructions consistent and uncomplicated?), clarity (is the question meaning clear?), assumptions (is the question based on correct assumptions or logic, or is it double barreled?), knowledge or memory (do respondents have the knowledge and cognitive ability required to answer?), sensitivity or bias (is the question, topic, or wording sensitive in nature?), and response categories (is the range of response options adequate?). An eighth category is used to catalog other types of problems. Each category includes and defines more specific problems. For example, survey items that lack clarity may be vague, use unfamiliar technical terms, have awkward syntax, or have ill-defined reference periods.
Our content analysis enumerated and categorized problems seen with the 30 items originally selected for pretesting and each subsequent revision. In sum, we pretested 59 items by using CI techniques and subjected them to content analysis. Interviewers worked independently, using separate forms to code problems with each of the items. They reviewed detailed summaries of residents' verbatim responses, interviewers' observations, and minutes from weekly team discussions to identify and assign problems to the most suitable QAS-99 category. Interrater reliability, calculated as the percentage of agreement between coders, was 92%. Coding disagreements occurred in deciding how problems were to be categorized, not in the identification of problems. We resolved these differences through discussion.
|
Results |
|---|
|
TOPAbstractMethodsResultsDiscussionReferences |
|---|
|
Item-Specific Problems
We fielded between one and five versions of each of the 30 items initially selected for CIs in an iterative process. The research team identified 61 problems specific to the 59 items that were pretested. Figure 1 shows a detailed example of the iterative process we used to refine an item.
|
|
Global Problems
In addition to item-specific problems, the team identified some problems that applied to multiple items across the survey. They did not include these problems in the tally of item-specific problems.
Problem With Response Categories: Accommodation
Problems with the response scale were of two types. The first was the tendency of some residents to accommodate perceived physical and environmental limitations. The second was the need for more precision than was offered by the 3-point scale.
In the first round of CIs, the research team tested seven items with four residents each. Of the discordant importance scale ratings and narrative responses in this round, content analysis revealed that four resulted from respondents' censoring of their importance ratings because of perceived limitations. For example, one item asked residents to rate how important it was "to do your favorite activities or hobbies." A participant who had been living in NHs for more than 3 years responded to the item saying it was not important. When prompted, he volunteered that although he enjoyed playing bingo, he wasn't able to play as often as he'd like because bingo was offered in the early evening after staff put him to bed. Clearly the resident suppressed his preferences and did not question the need for staff to transfer him to bed by 4 p.m., or the scheduling of bingo. Instead, he had accommodated to the restrictions in the NH environment.
Through responses like these, the research team observed that, when residents were answering some items, the residents' cognitive processes included an assessment of physical ability to engage in the activity and perceived limitations in the NH environment. After making these kinds of mental calculations, some residents with perceived limitations would rate a strongly preferred activity as being not important, whereas others would still rate it as being somewhat important or very important. As a result, the rating choices did not necessarily reflect an item's actual importance to every respondent. This accommodation was particularly problematic given that the intent of the items is to help staff identify care planning needs. Facilities might be expected to identify and care plan to avoid barriers that prevent residents from realizing their preferences.
In response, the research team introduced an additional and somewhat unique response option, "important, but can't do–no choice," to simplify the cognitive task. This option was intended to promote dialogue about perceived physical and environmental barriers that residents believed limited their ability to exercise their preferences.
Problem With Response Categories: Need for Greater Precision in the Response Scale
CIs suggested an additional source of discordance between resident ratings and their narrative responses to the probes. Residents who selected the "somewhat important" rating gave verbal narratives that were seldom neutral. Some narratives reflected a positive preference and others reflected a negative one. For example, when asked about the importance of choosing a time to be awakened in the morning, a resident, after some hesitation, rated the item as somewhat important. As part of his response to probing, he commented, "I like to set my own hours and I'd be very upset if that were taken away from me."
This overlap suggested that a 5-point importance scale might better represent resident's true preferences. In order to improve the precision and ease of answering the "somewhat important" option, we returned to the field to test two 5-point importance scales. Both metrics included the options of very important, somewhat important, and important, but can't do–no choice. They differed for the unimportant responses choices. Metric 1 used somewhat unimportant and not very important whereas Metric 2 used not very important and not important at all. CIs indicated that the "somewhat unimportant" option confused the residents. Therefore, we used Metric 2.
We next compared the 4-point (very important, somewhat important, not important, and important, but can't do–no choice) and 5-point importance scale. Interviews revealed that the 5-point importance scale was associated with greater concordance between residents' ratings and their narrative responses.
Problems With Assumptions
Some residents had difficulty selecting a single response for items relating to the scheduling of daily routines. Their narrative comments revealed that scheduling preferences may vary depending on how they feel or on the events of a particular day. For example, resident preferences for being awakened in the morning are highly individualized and variable. When asked about the importance of choosing a time to be awakened, a resident who rated the item as somewhat important commented that "it's a day-to-day thing," dependent on "how you feel or when you went to bed." Presumably, having the flexibility to choose a time to be awakened would be very important to the resident during periods of illness or after having stayed up late, and perhaps less important on other days.
What was wanted was choice and flexibility, approaches that are critical to the provision of person-centered care. Because choice and flexibility are qualities of care that should be universally available to all residents and not premised on a preendorsed item, the research team did not include a specific preference item for flexibility in the PAT. Likewise, items related to kindness and respect, although appropriate for satisfaction surveys, were viewed by the research team as fundamental elements that should be universally available and not require a resident to indicate they are important to them.
Item Selection for Further Pilot Testing
After 16 rounds of CIs, the team selected 24 items for further pilot testing. Table 3 shows the 24 items, 14 of which relate to 13 constructs included in the original CR section. As the table shows, we retained at least 1 item from each of seven quality-of-life domains. The table also shows 6 of the 30 original topics that, despite efforts to revise or replace, were not easily understood by residents during CIs and were therefore rejected.
|
|
Discussion |
|---|
|
TOPAbstractMethodsResultsDiscussionReferences |
|---|
In this study we used CIs to improve both the content and structure of the PAT. CIs allowed us, the research team, to identify many item-specific and global problems that would have potentially decreased the usefulness of the survey for eliciting preferences. Our team continuously revisited the logic underlying the survey items, and we made modifications that enhanced their clarity, relevance, sensitivity, and scope. The team also found that residents often selected response options based on accommodation to perceived barriers, and that response sets with insufficient options complicate the response process. These kinds of limitations, if not addressed, could limit the utility of preference surveys for informing the design of individualized care plans.
CI methods offer an important, systematic approach for gaining insights into the NH resident's experience, and for testing and revising individual items before conducting conventional survey testing. Although the CI process can be time consuming and labor intensive, its use in a small sample of residents leads to the development of more accurate and less demanding surveys for the larger population.
Limitations
We limited our CIs to residents who were cognitively intact; however, NH populations have high rates of cognitive impairment. Individuals with moderate, and in some cases even severe, cognitive impairment are able to respond to questions about their preferences and quality of life (Brod, Stewart, Sands, & Walton, 1999; Feinberg & Whitlatch, 2001; Van Haitsma, 2000; Whitlatch et al., 2005). Nevertheless, the need to articulate thoughts during the cognitive interview process would be burdensome for this population. We therefore focused on cognitively intact residents, reasoning that if an item was unclear to those with intact cognitive function, it would not perform well across the population. We used participants' feedback to make the PAT more accommodating to the NH environment, more sensitive to physical challenges, and, we hope, more easily understood by residents.
The sample of Department of Veterans Affairs (VA) NH residents participating in the CI process was predominantly male. However, we selected domains and related candidate and replacement items for the PAT on the basis of a literature review pertaining to general NH research, which is predominantly conducted in non-VA facilities. In those facilities, female residents greatly outnumber male residents. Furthermore, we considered all resident responses valuable at this stage of survey development, and we eliminated no items that were easily understood and answered because they were rated as being not important by our NH sample. Rather, our qualitative study focused on refining items to maximize their clarity and hence usefulness for communicating with all residents about their preferences. This goal, which we pursued by probing the cognitive processes that residents use to respond to survey items, should not be expected to be dependent on gender.
Future Research
After CI methods are used to refine items and surveys, next steps may include pilot testing for response rates and reliability in larger and more diverse samples. Next steps in the development of the PAT will include test–retest situations with cognitively intact and impaired, male and female, NH residents living in geographically diverse VA and non-VA facilities. Researchers will conduct these tests before final recommendation to include the PAT in the MDS or to use PAT as a stand-alone instrument.
If the PAT performs well, researchers should examine whether the interview initiates conversations between caregivers and care recipients, whether it facilitates individualized care planning, and whether it conveys to residents that the expression of preferences is valued and will be honored whenever possible. Given that preferences may change in response to development, life events, or reflection (Carpenter et al., 2000; Kane & Degenholtz, 1997), future studies should also examine the use of the PAT to elicit each resident's preferences over time.
Conclusions
This study illustrates that CI techniques can be used successfully to refine surveys intended for NH populations. Others interested in adapting an existing survey or creating and testing new surveys for frail populations can consider the potential application of this method for informing their work. In the current study, CI findings challenged commonly held assumptions about language use, item length, and the simplicity of response sets. The results revealed accommodation and day-to-day fluctuation of wants as elements of resident self-report, and the importance of developing precise response options that are easily mapped to the resident's experience.
|
Footnotes |
|---|
We wish to thank members of the Validation Panel for their contribution to this project: Dan Berlowitz, Boston University; Barbara Bowers, University of Wisconsin; Richard Della Penna, Kaiser Permanente Aging Network; Marcy Harris, Mayo Clinic; Ira Katz, University of Pennsylvania; Paul Katz, University of Rochester; Rosemary Lubinski, University of Buffalo; David Mehr, University of Missouri; Vince Mor, Brown University; Christine Ann Mueller, University of Minnesota; Patricia Parmelee, Emory University; Margaret Schenkman, University of Colorado; Neville Strumpf, University of Pennsylvania; Eric Tangalos, Mayo Clinic; Christie Teigland, New York Association of Homes and Services for the Aging; Sheryl Zimmerman, University of North Carolina at Chapel Hill.
We also thank Julie Brown of the survey research group at RAND Corporation, the decision editor, and two anonymous reviewers for their helpful comments and suggestions.
The research reported here was supported by the Department of Veterans Affairs, Veterans Health Administration, Health Services Research and Development Service SDR 03-217. The views expressed in this article are those of the authors and do not necessarily represent the views of the Department of Veterans Affairs.
1 Veterans Administration Greater Los Angeles Healthcare System HSR&D Center of Excellence and Geriatrics Research and Clinical Care.
2 RAND Corporation, Santa Monica, CA.
3 School of Nursing, University of California, Los Angeles.
4 Department of Health Care Policy, Harvard Medical School, Boston, MA.
5 Borun Center, University of California, Los Angeles and Jewish Homes for the Aging.
Decision Editor: William J. McAuley, PhD
Received for publication January 31, 2007. Accepted for publication June 7, 2007.
|
References |
|---|
|
TOPAbstractMethodsResultsDiscussionReferences |
|---|
| ||||||||||||||||||||||||
| HOME | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|---|
