Building Research Capacity| Volume 42, ISSUE 5, P307-318, June 2019

Download started.


Researching the Appropriateness of Care in the Complementary and Integrative Health Professions Part 3: Designing Instruments With Patient Input



      The purpose of this article is to describe how we designed patient survey instruments to ensure that patient data about preferences and experience could be included in appropriateness decisions. These actions were part of a project that examined the appropriateness of spinal manipulation and mobilization for chronic low back pain and chronic neck pain.


      We conducted focus groups, cognitive interviews, a literature review of measures in prior chiropractic and complementary and integrative health research, and a pilot study to develop questionnaires of patient preferences, experiences, values, and beliefs.


      Questionnaires were administered online to 2024 individuals from 125 chiropractic clinics. The survey included 3 long questionnaires and 5 shorter ones. All were administered online. The baseline items had 2 questionnaires that respondents could complete in different sittings. Respondents completed shorter biweekly follow-ups every 2 weeks and a final questionnaire at 3 months. The 2 initial questionnaires had 81 and 140 items, the 5 biweekly follow-up questionnaires had 37 items each, and the endline questionnaire contained 121 items. Participants generally responded positively to the survey items, and 91% of the patients who completed a baseline questionnaire completed the endpoint survey 3 months later. We used “legacy” measures, and we also adapted measures and developed new measures for this study. Preliminary assessment of reliability and validity for a newly developed scale about coping behaviors indicates that the items work well together in a scale.


      This article documents the challenges and the efforts involved in designing data collection tools to facilitate the inclusion of patient data into appropriateness decisions.

      Key Indexing Terms

      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Journal of Manipulative & Physiological Therapeutics
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Patient-Centered Outcomes Research Institute (PCORI)
        (Available at:)
        Date accessed: November 19, 2018
        • Coulter I
        • Herman PM
        • Ryan GG
        • Hays RD
        • Hilton LG
        • Whitley MD
        Researching the appropriateness of care in the complementary and integrative health professions: part I.
        J Manipulative Physiol Ther. 2018; 41: 800-806
        • Coulter I
        • Elfenbaum P
        • Jain S
        • Jonas W
        SEaRCH™ expert panel process: streamlining the link between evidence and practice.
        BMC Res Notes. 2016; 9: 16
        • Fitch K
        • Berstein SJ
        • Aguilar MD
        • et al.
        The RAND/UCLA Appropriateness Method User’s Manual.
        RAND Corporation, Santa Monica, CA2001
        • Coulter I
        • Whitley M
        • Hurwitz E
        • Vernon H
        • Shekelle P
        • Herman PM
        Determining the Appropriateness of Spinal Manipulation and Mobilization for Chronic Low Back Pain: Indications and Ratings by a Multidisciplinary Expert Panel.
        RAND Corp, Santa Monica, CA2018
        • Coulter I
        • Whitley M
        • Vernon H
        • Hurwitz E
        • Shekelle P
        • Herman PM
        Determining the Appropriateness of Spinal Manipulation and Mobilization for Chronic Neck Pain: Indications and Ratings by a Multidisciplinary Expert Panel.
        RAND Corp, Santa Monica, CA2018
        • Fung CH
        • Hays RD
        Prospects and challenges in using patient-reported outcomes in clinical practice.
        Qual Life Res. 2008; 17: 1297-1302
        • Deshpande PR
        • Rajan S
        • Sudeepthi BL
        • Abdul Nazir CP
        Patient-reported outcomes: a new era in clinical research.
        Pers Clin Res. 2011; 2: 137-144
        • Bitton A
        • Onega T
        • Tosteson ANA
        • Haas JS
        Toward a better understanding of patient-reported outcomes in clinical practice.
        Am J Manag Care. 2014; 20: 281-283
        • Snyder CF
        • Aaronson NK
        • Choucair AK
        • et al.
        Implementing patient-reported outcomes assessment in clinical practice: a review of the options and considerations.
        Qual Life Res. 2012; 21: 1305-1314
        • Khorsan R
        • Coulter ID
        • Hawk C
        • Choate CG
        Measures in chiropractic research: choosing patient-based outcome assessments.
        J Manipulative Physiol Ther. 2008; 31: 355-375
        • Wood-Dauphinee S
        Assessing quality of life in clinical research: from where have we come and where are we going?.
        J Clin Epidemiol. 1999; 52: 355-363
        • Cherepanov D
        • Hays R
        Health and quality-of-life-outcomes: the role of patient-reported measures.
        in: Outcomes Measurement in the Human Services: Cross-Cultural Issues and Methods in the Era of Health Reform. 2nd ed. NASM Press, Washington, DC2011
        • Ganz P
        • Hays RD
        • Kaplan RM
        • Litwin MS
        Measuring health-related quality of life and other outcomes.
        in: Kominski GG Changing the U.S. Health Care System. 4th ed. Jossey-Bass, San Francisco, CA2014: 307-341
        • Shahgholi L
        • Yost KJ
        • Carter RE
        • et al.
        Correlation of the Patient Reported Outcomes Measurement Information System with legacy outcomes measures in assessment of response to lumbar transforaminal epidural steroid injections.
        Am J Neuroradiol. 2015; 36: 594
        • Sikorskii A
        • Victorson D
        • O’Connor P
        • et al.
        PROMIS and legacy measures compared in a supportive care intervention for breast cancer patients and caregivers: experience from a randomized trial.
        Psychooncology. 2018; 27: 2265-2273
        • Clover K
        • Lambert SD
        • Oldmeadow C
        • et al.
        PROMIS depression measures perform similarly to legacy measures relative to a structured diagnostic interview for depression in cancer patients.
        Qual Life Res. 2018; 27: 1357-1367
        • Ryan GW
        • Bernard HR
        Techniques to identify themes.
        Field Methods. 2003; 15: 85-109
        • Spiegel BMR
        • Hays RD
        • Bolus R
        • et al.
        Development of the NIH Patient-Reported Outcomes Measurement Information System (PROMIS) Gastrointestinal Symptom Scales.
        Am J Gastroenterol. 2014; 109: 1804-1814
        • Maida CA
        • Marcus M
        • Hays RD
        • et al.
        Child and adolescent perceptions of oral health over the life course.
        Qual Life Res. 2015; 24: 2739-2751
        • Fairbank J
        • Couper J
        • Davies J
        • O’Brien J
        The Oswestry Low Back Pain Disability Questionnaire.
        Physiotherapy. 1980; 66: 271-273
        • Fairbank JC
        • Pynsent PB
        The Oswestry Disability Index.
        Spine. 2000; 25: 2940-2953
        • Vernon H
        • Mior S
        The Neck Disability Index: a study of reliability and validity.
        J Manipulative Physiol Ther. 1991; 14: 409-415
        • National Institutes of Health (NIH)
        Patient-Reported Outcomes Measurement Information Systems (PROMIS®).
        (Available at:)
        Date accessed: November 19, 2018
        • Jensen MP
        • Karoly P
        • Braver S
        The measurement of clinical pain intensity: a comparison of six methods.
        Pain. 1986; 27: 117-126
        • Jensen MP
        • Turner JA
        • Romano JM
        • Lawler BK
        Relationship of pain-specific beliefs to chronic pain adjustment.
        Pain. 1994; 57: 301-309
        • Anderson KO
        • Dowds BN
        • Pelletz RE
        • Edwards WT
        • Peeters-Asdourian C
        Development and initial validation of a scale to measure self-efficacy beliefs in patients with chronic pain.
        Pain. 1995; 63: 77-83
        • Morone NE
        • Greco CM
        • Moore CG
        • et al.
        A mind-body program for older adults with chronic low back pain: a randomized clinical trial.
        JAMA Intern Med. 2016; 176: 329-337
        • Shpaner M
        • Kelly C
        • Lieberman G
        • et al.
        Unlearning chronic pain: a randomized controlled trial to investigate changes in intrinsic brain connectivity following cognitive behavioral therapy.
        Neuroimage Clin. 2014; 5: 365-376
        • Williams KA
        • Petronis J
        • Smith D
        • et al.
        Effect of Iyengar yoga therapy for chronic low back pain.
        Pain. 2005; 115: 107-117
        • Devilly GJ
        • Borkovec TD
        Psychometric properties of the credibility/expectancy questionnaire.
        J Behav Ther Exp Psychiatry. 2000; 31: 73-86
      1. The Center for Epidemiologic Studies Depression Scale Revised.
        (Available at:) (Accessed October 23, 2018)
        • Hays RD
        • Weech-Maldonado R
        • Teresi JA
        • Wallace SP
        • Stewart AL
        Commentary: copyright restrictions versus open access to survey instruments.
        Med Care. 2018; 56: 107-110
        • Fowler FJ
        Improving Survey Questions: Design and Evaluation.
        SAGE Publications, Thousand Oaks, CA1995
      2. Johnson RL, Morgan GB. Survey Scales: A Guide to Development, Analysis, and Reporting. New York, NY: Guilford Publications; 2016.

      3. Willis GB, Artino AR Jr. What do our respondents think we’re asking? Using cognitive interviewing to improve medical education surveys. J Grad Med Educ 2013;5(3):353-356.

        • Barry LC
        • Kerns RD
        • Guo Z
        • Duong BD
        • Iannone LP
        • Reid MC
        Identification of strategies used to cope with chronic pain in older persons receiving primary care from a Veterans Affairs Medical Center.
        J Am Geriatr Soc. 2004; 52: 950-956
        • Fleishman JA
        • Sherbourne CD
        • Cleary PD
        • Wu AW
        • Crystal S
        • Hays RD
        Patterns of coping among persons with HIV infection: configurations, correlates, and change.
        Am J Community Psychol. 2003; 32: 187-204
        • Deyo RA
        • Dworkin SF
        • Amtmann D
        • et al.
        Report of the NIH Task Force on research standards for chronic low back pain.
        Phys Ther. 2015; 95: e1-e18
        • Singleton Jr., R
        • Straits BC
        Approaches to Social Research. 5th ed.
        Oxford University Press, Oxford, England2010
        • Cronbach LJ
        Coefficient alpha and the internal structure of tests.
        Psychometrika. 1951; 16: 297-334
        • Krosnick JA
        • Alwin DF
        An evaluation of a cognitive theory of response-order effects in survey measurement.
        Public Opin Q. 1987; 51: 201-219
        • Brook RH
        • Chassin MR
        • Fink A
        • Solomon DH
        • Kosecoff J
        • Park RE
        A method for the detailed assessment of the appropriateness of medical technologies.
        Int J Technol Assess Health Care. 2009; 2: 53-63
        • World Health Organization Regional Office for Europe
        Appropriateness in health care services: report on a WHO workshop, 23–25 March 2000. Koblenz, Germany.
        WHO Regional Office for Europe, Copenhagen, Denmark2000
        • Von Korff M
        • Ormel J
        • Keefe FJ
        • Dworkin SF
        Grading the severity of chronic pain.
        Pain. 1992; 50: 133-149
        • Linton SJ
        • Halldén K
        Can we screen for problematic back pain? A screening questionnaire for predicting outcome in acute and subacute back pain.
        Clin J Pain. 1998; 14: 209-215
        • Lorig K
        • Chastain RL
        • Ung E
        • Shoor S
        • Holman HR
        Development and evaluation of a scale to measure perceived self-efficacy in people with arthritis.
        Arthritis Rheum. 1989; 32: 37-44
        • Symonds T
        • Burton AK
        • Tillotson KM
        • Main CJ
        Do attitudes and beliefs influence work loss due to low back trouble?.
        Occup Med. 1996; 46: 25-32
        • Calin A
        • Garrett S
        • Whitelock H
        • et al.
        A new approach to defining functional ability in ankylosing spondylitis: the development of the Bath Ankylosing Spondylitis Functional Index.
        J Rheumatol. 1994; 21: 2281-2285
        • Beck AT
        • Epstein N
        • Brown G
        • Steer RA
        An inventory for measuring clinical anxiety: psychometric properties.
        J Consult Clin Psychol. 1988; 56: 893
        • Beck AT
        • Ward CH
        • Mendelson M
        • Mock J
        • Erbaugh J
        An inventory for measuring depression.
        Arch Gen Psychiatry. 1961; 4: 561-571
        • Borg G
        Perceived exertion as an indicator of somatic stress.
        Scand J Rehabil Med. 1970; 2: 92-98
        • Bolton JE
        • Breen AC
        The Bournemouth Questionnaire: a short-form comprehensive outcome measure. I. Psychometric properties in back pain patients.
        J Manipulative Physiol Ther. 1999; 22: 503-510
        • Radloff LS
        The CES-D scale: a self-report depression scale for research in the general population.
        Appl Psychol Meas. 1977; 1: 385-401
        • McCracken LM
        • Vowles KE
        • Eccleston C
        Acceptance of chronic pain: component analysis and a revised assessment method.
        Pain. 2004; 107: 159-166
        • Jensen MP
        • Turner JA
        • Romano JM
        • Strom SE
        The chronic pain coping inventory: development and preliminary validation.
        Pain. 1995; 60: 203-216
        • Rosenstiel AK
        • Keefe FJ
        The use of coping strategies in chronic low back pain patients: relationship to patient characteristics and current adjustment.
        Pain. 1983; 17: 33-44
        • Hudak PL
        • Amadio PC
        • Bombardier C
        • et al.
        Development of an upper extremity outcome measure: the DASH (disabilities of the arm, shoulder, and head).
        Am J Ind Med. 1996; 29: 602-608
        • Goldberg DP
        • Hillier VF
        A scaled version of the General Health Questionnaire.
        Psychol Med. 1979; 9: 139-145
        • Schwarzer R
        • Jerusalem M
        Generalized self-efficacy scale.
        in: Weinman J Wright S Johnston M Measures in Health Psychology: A User’s Portfolio. Causal and Control Beliefs. Windsor, UK: NFER-NELSON. 1995: 35-37
        • Fischer D
        • Stewart AL
        • Bloch DA
        • Lorig K
        • Laurent D
        • Holman H
        Capturing the patient’s view of change as a clinical outcome measure.
        JAMA. 1999; 282: 1157-1162
        • Jaeschke R
        • Singer J
        • Guyatt GH
        Measurement of health status: ascertaining the minimal clinically important difference.
        Control Clin Trials. 1989; 10: 407-415
        • Karsdorp PA
        • Vlaeyen JWS
        Goals matter: both achievement and pain-avoidance goals are associated with pain severity and disability in patients with low back and upper extremity pain.
        Pain. 2011; 152: 1382-1390
        • Centers for Disease Control and Prevention
        Health-related quality-of-life measures--United States, 1993.
        MMWR Morb Mortal Wkly Rep. 1995; 44: 195-200
        • Sullivan MJ
        • Adams H
        • Horan S
        • Maher D
        • Boland D
        • Gross R
        The role of perceived injustice in the experience of chronic pain and disability: scale development and validation.
        J Occup Rehabil. 2008; 18: 249-261
        • Beattie PF
        • Nelson R
        • Murphy DR
        Development and preliminary validation of the MedRisk instrument to measure patient satisfaction with chiropractic care.
        J Manipulative Physiol Ther. 2011; 34: 23-29
        • Wallston BS
        • Wallston KA
        • Kaplan GD
        • Maides SA
        Development and validation of the health locus of control (HLC) scale.
        J Consult Clin Psychol. 1976; 44: 580
        • Leak A
        • Cooper J
        • Dyer S
        • Williams K
        • Turner-Stokes L
        • Frank A
        The Northwick Park Neck Pain Questionnaire, devised to measure neck pain and disability.
        Rheumatology. 1994; 33: 469-474
        • Sullivan MJ
        • Bishop SR
        • Pivik J
        The Pain Catastrophizing Scale: development and validation.
        Psychol Assess. 1995; 7: 524
        • Chibnall JT
        • Tait RC
        The Pain Disability Index: factor structure and normative data.
        Arch Phys Med Rehabil. 1994; 75: 1082-1086
        • Nicholas MK
        The pain self-efficacy questionnaire: taking pain into account.
        Eur J Pain. 2007; 11: 153-163
        • Shaw WS
        • Woiszwillo MJ
        • Krupat E
        Further validation of the Patient-Practitioner Orientation Scale (PPOS) from recorded visits for back pain.
        Patient Educ Couns. 2012; 89: 288-291
        • Kim HS
        Collaborative decision making in nursing practice: a theoretical framework.
        in: Chinn P Advances in Nursing Theory Development. Aspen Systems, Rockville, MD1983: 271-283
        • Stratford P
        • Gill C
        • Westaway M
        • Binkley J
        Assessing disability and change on individual patients: a report of a patient specific measure.
        Physiother Canada. 1995; 47: 258-263
        • Leeuw M
        • Goossens ME
        • van Breukelen GJ
        • Boersma K
        • Vlaeyen JW
        Measuring perceived harmfulness of physical activities in patients with chronic low back pain: the Photograph Series of Daily Activities—short electronic version.
        J Pain. 2007; 8: 840-849
        • Wicksell RK
        • Lekander M
        • Sorjonen K
        • Olsson GL
        The Psychological Inflexibility in Pain Scale (PIPS)–Statistical properties and model fit of an instrument to assess change processes in pain related disability.
        Eur J Pain. 2010; 14: 771.e1-771.e14
        • Kopec JA
        • Esdaile JM
        • Abrahamowicz M
        • et al.
        The Quebec Back Pain Disability Scale. Measurement properties.
        Spine. 1995; 20: 341-352
        • Roter D
        • Larson S
        The Roter interaction analysis system (RIAS): utility and flexibility for analysis of medical interactions.
        Patient Educ Couns. 2002; 46: 243-251
        • Altmaier EM
        • Russell DW
        • Kao CF
        • Lehmann TR
        • Weinstein JN
        Role of self-efficacy in rehabilitation outcome among chronic low back pain patients.
        J Couns Psychol. 1993; 40: 335
        • Antonovsky A
        The structure and properties of the sense of coherence scale.
        Soc Sci Med. 1993; 36: 725-733
        • Brazier J
        • Usherwood T
        • Harper R
        • Thomas K
        Deriving a preference-based single index from the UK SF-36 Health Survey.
        J Clin Epidemiol. 1998; 51: 1115-1128
        • Roach KE
        • Budiman-Mak E
        • Songsiridej N
        • Lertratanakul Y
        Development of a shoulder pain and disability index.
        Arthritis Rheum. 1991; 4: 143-149
      4. Brewer B, Van Raalte J, Petitpas A, Sklar J, Ditmar T. A brief measure of adherence during sport injury rehabilitation sessions. J Appl Sport Psychol 1995;7(suppl):S44.

        • Vlaeyen JWS
        • Kole-Snijders AMJ
        • Boeren RGB
        • van Eek H
        Fear of movement/(re)injury in chronic low back pain and its relation to behavioral performance.
        Pain. 1995; 62: 363-372
        • Brown GK
        • Nicassio PM
        Development of a questionnaire for the assessment of active and passive coping strategies in chronic pain patients.
        Pain. 1987; 31: 53-64
        • Bellamy N
        • Buchanan WW
        • Goldsmith CH
        • Campbell J
        • Stitt LW
        Validation study of WOMAC: a health status instrument for measuring clinically important patient relevant outcomes to antirheumatic drug therapy in patients with osteoarthritis of the hip or knee.
        J Rheumatol. 1988; 15: 1833-1840
        • Horvath AO
        • Greenberg LS
        Development and validation of the Working Alliance Inventory.
        J Couns Psychol. 1989; 36: 223
        • Zung WW
        A self-rating depression scale.
        Arch Gen Psychiatry. 1965; 12: 63-70