program evaluation research definition

• Skip to main content
• Skip to primary sidebar
• Skip to footer
• QuestionPro

• Solutions Industries Gaming Automotive Sports and events Education Government Travel & Hospitality Financial Services Healthcare Cannabis Technology Use Case AskWhy Communities Audience Contactless surveys Mobile LivePolls Member Experience GDPR Positive People Science 360 Feedback Surveys
• Resources Blog eBooks Survey Templates Case Studies Training Help center

Home Market Research

Evaluation Research: Definition, Methods and Examples

Content Index

• What is evaluation research
• Why do evaluation research

Quantitative methods

Qualitative methods.

• Process evaluation research question examples
• Outcome evaluation research question examples

What is evaluation research?

Evaluation research, also known as program evaluation, refers to research purpose instead of a specific method. Evaluation research is the systematic assessment of the worth or merit of time, money, effort and resources spent in order to achieve a goal.

Evaluation research is closely related to but slightly different from more conventional social research . It uses many of the same methods used in traditional social research, but because it takes place within an organizational context, it requires team skills, interpersonal skills, management skills, political smartness, and other research skills that social research does not need much. Evaluation research also requires one to keep in mind the interests of the stakeholders.

Evaluation research is a type of applied research, and so it is intended to have some real-world effect.  Many methods like surveys and experiments can be used to do evaluation research. The process of evaluation research consisting of data analysis and reporting is a rigorous, systematic process that involves collecting data about organizations, processes, projects, services, and/or resources. Evaluation research enhances knowledge and decision-making, and leads to practical applications.

LEARN ABOUT: Action Research

Why do evaluation research?

The common goal of most evaluations is to extract meaningful information from the audience and provide valuable insights to evaluators such as sponsors, donors, client-groups, administrators, staff, and other relevant constituencies. Most often, feedback is perceived value as useful if it helps in decision-making. However, evaluation research does not always create an impact that can be applied anywhere else, sometimes they fail to influence short-term decisions. It is also equally true that initially, it might seem to not have any influence, but can have a delayed impact when the situation is more favorable. In spite of this, there is a general agreement that the major goal of evaluation research should be to improve decision-making through the systematic utilization of measurable feedback.

Below are some of the benefits of evaluation research

• Gain insights about a project or program and its operations

Evaluation Research lets you understand what works and what doesn’t, where we were, where we are and where we are headed towards. You can find out the areas of improvement and identify strengths. So, it will help you to figure out what do you need to focus more on and if there are any threats to your business. You can also find out if there are currently hidden sectors in the market that are yet untapped.

• Improve practice

It is essential to gauge your past performance and understand what went wrong in order to deliver better services to your customers. Unless it is a two-way communication, there is no way to improve on what you have to offer. Evaluation research gives an opportunity to your employees and customers to express how they feel and if there’s anything they would like to change. It also lets you modify or adopt a practice such that it increases the chances of success.

• Assess the effects

After evaluating the efforts, you can see how well you are meeting objectives and targets. Evaluations let you measure if the intended benefits are really reaching the targeted audience and if yes, then how effectively.

• Build capacity

Evaluations help you to analyze the demand pattern and predict if you will need more funds, upgrade skills and improve the efficiency of operations. It lets you find the gaps in the production to delivery chain and possible ways to fill them.

Methods of evaluation research

All market research methods involve collecting and analyzing the data, making decisions about the validity of the information and deriving relevant inferences from it. Evaluation research comprises of planning, conducting and analyzing the results which include the use of data collection techniques and applying statistical methods.

Some of the evaluation methods which are quite popular are input measurement, output or performance measurement, impact or outcomes assessment, quality assessment, process evaluation, benchmarking, standards, cost analysis, organizational effectiveness, program evaluation methods, and LIS-centered methods. There are also a few types of evaluations that do not always result in a meaningful assessment such as descriptive studies, formative evaluations, and implementation analysis. Evaluation research is more about information-processing and feedback functions of evaluation.

These methods can be broadly classified as quantitative and qualitative methods.

The outcome of the quantitative research methods is an answer to the questions below and is used to measure anything tangible.

• Who was involved?
• What were the outcomes?
• What was the price?

The best way to collect quantitative data is through surveys , questionnaires , and polls . You can also create pre-tests and post-tests, review existing documents and databases or gather clinical data.

Surveys are used to gather opinions, feedback or ideas of your employees or customers and consist of various question types . They can be conducted by a person face-to-face or by telephone, by mail, or online. Online surveys do not require the intervention of any human and are far more efficient and practical. You can see the survey results on dashboard of research tools and dig deeper using filter criteria based on various factors such as age, gender, location, etc. You can also keep survey logic such as branching, quotas, chain survey, looping, etc in the survey questions and reduce the time to both create and respond to the donor survey . You can also generate a number of reports that involve statistical formulae and present data that can be readily absorbed in the meetings. To learn more about how research tool works and whether it is suitable for you, sign up for a free account now.

Create a free account!

Quantitative data measure the depth and breadth of an initiative, for instance, the number of people who participated in the non-profit event, the number of people who enrolled for a new course at the university. Quantitative data collected before and after a program can show its results and impact.

The accuracy of quantitative data to be used for evaluation research depends on how well the sample represents the population, the ease of analysis, and their consistency. Quantitative methods can fail if the questions are not framed correctly and not distributed to the right audience. Also, quantitative data do not provide an understanding of the context and may not be apt for complex issues.

Learn more: Quantitative Market Research: The Complete Guide

Qualitative research methods are used where quantitative methods cannot solve the research problem , i.e. they are used to measure intangible values. They answer questions such as

• What is the value added?
• How satisfied are you with our service?
• How likely are you to recommend us to your friends?
• What will improve your experience?

LEARN ABOUT: Qualitative Interview

Qualitative data is collected through observation, interviews, case studies, and focus groups. The steps for creating a qualitative study involve examining, comparing and contrasting, and understanding patterns. Analysts conclude after identification of themes, clustering similar data, and finally reducing to points that make sense.

Observations may help explain behaviors as well as the social context that is generally not discovered by quantitative methods. Observations of behavior and body language can be done by watching a participant, recording audio or video. Structured interviews can be conducted with people alone or in a group under controlled conditions, or they may be asked open-ended qualitative research questions . Qualitative research methods are also used to understand a person’s perceptions and motivations.

LEARN ABOUT:  Social Communication Questionnaire

The strength of this method is that group discussion can provide ideas and stimulate memories with topics cascading as discussion occurs. The accuracy of qualitative data depends on how well contextual data explains complex issues and complements quantitative data. It helps get the answer of “why” and “how”, after getting an answer to “what”. The limitations of qualitative data for evaluation research are that they are subjective, time-consuming, costly and difficult to analyze and interpret.

Learn more: Qualitative Market Research: The Complete Guide

Survey software can be used for both the evaluation research methods. You can use above sample questions for evaluation research and send a survey in minutes using research software. Using a tool for research simplifies the process right from creating a survey, importing contacts, distributing the survey and generating reports that aid in research.

Examples of evaluation research

Evaluation research questions lay the foundation of a successful evaluation. They define the topics that will be evaluated. Keeping evaluation questions ready not only saves time and money, but also makes it easier to decide what data to collect, how to analyze it, and how to report it.

Evaluation research questions must be developed and agreed on in the planning stage, however, ready-made research templates can also be used.

Process evaluation research question examples:

• How often do you use our product in a day?
• Were approvals taken from all stakeholders?
• Can you report the issue from the system?
• Can you submit the feedback from the system?
• Was each task done as per the standard operating procedure?
• What were the barriers to the implementation of each task?
• Were any improvement areas discovered?

Outcome evaluation research question examples:

• How satisfied are you with our product?
• Did the program produce intended outcomes?
• What were the unintended outcomes?
• Has the program increased the knowledge of participants?
• Were the participants of the program employable before the course started?
• Do participants of the program have the skills to find a job after the course ended?
• Is the knowledge of participants better compared to those who did not participate in the program?

MORE LIKE THIS

Participant Engagement: Strategies + Improving Interaction

Sep 12, 2024

Employee Recognition Programs: A Complete Guide

Sep 11, 2024

A guide to conducting agile qualitative research for rapid insights with Digsite 

Cultural Insights: What it is, Importance + How to Collect?

Sep 10, 2024

Other categories

• Academic Research
• Artificial Intelligence
• Assessments
• Brand Awareness
• Case Studies
• Communities
• Consumer Insights
• Customer effort score
• Customer Engagement
• Customer Experience
• Customer Loyalty
• Customer Research
• Customer Satisfaction
• Employee Benefits
• Employee Engagement
• Employee Retention
• Friday Five
• General Data Protection Regulation
• Insights Hub
• Life@QuestionPro
• Market Research
• Mobile diaries
• Mobile Surveys
• New Features
• Online Communities
• Question Types
• Questionnaire
• QuestionPro Products
• Release Notes
• Research Tools and Apps
• Revenue at Risk
• Survey Templates
• Training Tips
• Tuesday CX Thoughts (TCXT)
• Uncategorized
• What’s Coming Up
• Workforce Intelligence

Contact Us (315) 303-2040

Evaluation Market Research

Discover expert insights, methodologies, and strategies to make informed business and stay ahead in your industry with evaluation research.

Request a Quote Read the Blog

• Market Research Company Blog

What is Evaluation Research? + [Methods & Examples]

by Emily Taylor

Posted at: 2/20/2023 1:30 PM

Every business and organization has goals. 

But, how do you know if the time, money, and resources spent on strategies to achieve these goals are working?

Or, if they’re even worth it? 

Evaluation research is a great way to answer these common questions as it measures how effective a specific program or strategy is.

In this post, we’ll cover what evaluation research is, how to conduct it, the benefits of doing so, and more.

Article Contents

• Definition of evaluation research
• The purpose of program evaluation
• Evaluation research advantages and disadvantages
• Research evaluation methods
• Examples and types of evaluation research
• Evaluation research questions

Evaluation Research: Definition

Evaluation research, also known as program evaluation, is a systematic analysis that evaluates whether a program or strategy is worth the effort, time, money, and resources spent to achieve a goal. 

Based on the project’s objectives, the study may target different audiences such as: 

• Stakeholders
• Prospective customers
• Board members

The feedback gathered from program evaluation research is used to validate whether something should continue or be changed in any way to better meet organizational goals.  

The Purpose of Program Evaluation

The main purpose of evaluation research is to understand whether or not a process or strategy has delivered the desired results. 

It is especially helpful when launching new products, services, or concepts.

That’s because research program evaluation allows you to gather feedback from target audiences to learn what is working and what needs improvement. 

It is a vehicle for hearing people’s experiences with your new concept to gauge whether it is the right fit for the intended audience.

And with data-driven companies being 23 times more likely to acquire customers, it seems like a no-brainer.

As a result of evaluation research, organizations can better build a program or solution that provides audiences with exactly what they need.

Better yet, it’s done without wasting time and money figuring out new iterations before landing on the final product.

Evaluation Research Advantages & Disadvantages

In this section, our market research company dives more into the benefits and drawbacks of conducting research evaluation methods.

Understanding these pros and cons will help determine if it’s right for your business.

Advantages of Evaluation Research

In many instances, the pros of program evaluation outweigh the cons.

It is an effective tool for data-driven decision-making and sets organizations on a clear path to success.

Here are just a few of the many benefits of conducting research evaluation methods.

Justifies the time, money, and resources spent

First, evaluation research helps justify all of the resources spent on a program or strategy. 

Without evaluation research, it can be difficult to promote the continuation of a costly or time-intensive activity with no evidence it’s working. 

Rather than relying on opinions and gut reactions about the effectiveness of a program or strategy, evaluation research measures levels of effectiveness through data collected. 

Identifies unknown negative or positive impacts of a strategy

Second, program research helps users better understand how projects are carried out, who helps them come to fruition, who is affected, and more. 

These finer details shed light on how a program or strategy affects all facets of an organization.

As a result, you may learn there are unrealized effects that surprise you and your decision-makers.

Helps organizations improve

The research can highlight areas of strengths (i.e., factors of the program/strategy that should not be changed) and weaknesses (i.e., factors of the programs/strategy that could be improved).

Disadvantages of Evaluation Research

Despite its many advantages, there are still limitations and drawbacks to evaluation research.

Here are a few challenges to keep in mind before moving forward.

It can be costly

The cost of market research varies based on methodology, audience type, incentives, and more.

For instance, a focus group will be more expensive than an online survey.

Though, I’ll also make the argument that conducting evaluation research can save brands money down the line from investing in something that is a dud.

Poor memory recall

Many research evaluation methods are dependent on feedback from customers, employees, and other audiences. 

If the study is not conducted right after a process or strategy is implemented, it can be harder for these audiences to remember their true opinions and feelings on the matter.

Therefore, the data might be less accurate because of the lapse in time and memory.

Research Evaluation Methods

Evaluation research can include a mix of qualitative and quantitative methods depending on your objectives. 

A market research company , like Drive Research , can design an approach to best meet your goals, objectives, and budget for a successful study.

Below we share different approaches to evaluation research.

But, here is a quick graphic that explains the main differences between qualitative and quantitative research methodologies .

Quantitative Research Methods

Quantitative evaluation research aims to measure audience feedback.

Metrics quantitative market research often measures include:

• Level of impact
• Level of awareness
• Level of satisfaction
• Level of perception
• Expected usage
• Usage of competitors

In addition to other metrics to gauge the success of a program or strategy. 

This type of evaluation research can be done through online surveys or phone surveys. 

Online surveys

Perhaps the most common form of quantitative research , online surveys are extremely effective for gathering feedback. 

They are commonly used for evaluation research because they offer quick, cost-effective, and actionable insights.

Typically, the survey is conducted by a third-party online survey company to ensure anonymity and limit bias from the respondents. 

The market research firm develops the survey, conducts fieldwork, and creates a report based on the results.

For instance, here is the online survey process followed by Drive Research when conducting program evaluations for our clients.

Phone surveys

Another way to conduct evaluation research is with phone surveys .

This type of market research allows trained interviewers to have one-on-one conversations with your target audience. 

Oftentimes they range from 15 to 30-minute discussions to gather enough information and feedback. 

The benefit of phone surveys for program evaluation research is that interviewers can ask respondents to explain their answers in more detail. 

Whereas, an online survey is limited to multiple-choice questions with pre-determined answer options (with the addition of a few open ends). 

Though, online surveys are much faster and more cost-effective to complete.

Recommended Reading: What is the Most Cost-Effective Market Research Methodology?

Qualitative Research Methods

Qualitative evaluation research aims to explore audience feedback.

Factors quantitative market research often evaluates include:

• Areas of satisfaction
• Areas of weaknesses
• Recommendations

This type of exploratory evaluation research can be completed through in-depth interviews or focus groups.

It involves working with a qualitative recruiting company to recruit specific types of people for the research, developing a specific line of questioning, and then summarizing the results to ensure anonymity.

For instance, here is the process Drive Research follows when recruiting people to participate in evaluation research. 

Focus groups

If you are considering conducting qualitative evaluation research, it’s likely that focus groups are your top methodology of choice.

Focus groups are a great way to collect feedback from targeted audiences all at once.

It is also a helpful methodology for showing product markups, logo designs, commercials, and more.

Though, a great alternative to traditional focus groups is online focus groups .

Remote focus groups can reduce the costs of evaluation research because it eliminates many of the fees associated with in-person groups.

For instance, there are no facility rental fees.

Plus, recruiting participants is cheaper because you can cast a wider net being that they can join an online forum from anywhere in the country.

In-depth interviews (IDIs)

Similar to focus groups, in-depth interviews gather tremendous amounts of information and feedback from target consumers. 

In this setting though, interviewers speak with participants one-on-one, rather than in a group. 

This level of attention allows interviewers to expand on more areas of what satisfies and dissatisfies someone about a product, service, or program. 

Additionally, it eliminates group bias in evaluation research.

This is because participants are more comfortable providing honest opinions without being intimidated by others in a focus group.

Examples and Types of Evaluation Research

There are different types of evaluation research based on the business and audience type.

Most commonly it is carried out for product concepts, marketing strategies, and programs.

We share a few examples of each below.

Product Evaluation Research Example 

Each year, 95 percent of new products introduced to the market fail. 

Therefore market research for new product development is critical in determining what could deter the success of a concept before it reaches shelves.

Lego is a great example of a brand using evaluation research for new product concepts.

In 2011 they learned 90% of their buyers were boys. 

Although boys were not their sole target demographic, the brand had more products that were appealing to this audience such as Star Wars and superheroes. 

To grow its audience, Lego conducted evaluation research to determine what topics and themes would entice female buyers.

With this insight, Lego launched Lego Friends. It included more details and features girls were looking for. 

Marketing Evaluation Research Example 

Marketing evaluation research or campaign evaluation surveys is a technique used to measure the effectiveness of advertising and marketing strategies. 

An example of this would be surveying a target audience before and after launching a paid social media campaign. 

Brands can determine if factors such as awareness, perception, and likelihood to purchase have changed due to the advertisements. 

Recommended Reading: Advertising Testing with Market Research

Process Evaluation Research Example

Process evaluations are commonly used to understand the implementation of a new program.

It helps decision-makers evaluate how a program’s goal or outcome was achieved. 

Additionally, process evaluation research quantifies how often the program was used, who benefited from the program, the resources used to implement the new process, any problems encountered, and more.

Examples of programs and processes where evaluation research is beneficial are:

• Customer loyalty programs
• Client referral programs
• Customer retention programs
• Workplace wellness programs
• Orientation of new employees
• Employee buddy programs 

Evaluation Research Questions

Evaluation research design sets the tone for a successful study.

It is important to ask the right questions in order to achieve the intended results. 

Product evaluation research questions include:

• How appealing is the following product concept?
• If available in a store near you, how likely are you to purchase [product]?
• Which of the following packaging types do you prefer?
• Which of the following [colors, flavors, sizes, etc.] would you be most interested in purchasing?

Marketing evaluation research questions include:

• Please rate your level of awareness for [Brand].
• What is your perception of [Brand]?
• Do you remember seeing advertisements for [Brand] in the past 3 months?
• Where did you see or hear the advertising for [Brand]? ie. Facebook, TV, radio, etc.
• How likely are you to make a purchase from [Brand]?

Process evaluation research questions include:

• Please rate your level of satisfaction with [Process].
• Please explain why you provided [Rating].
• What barriers existed to implementing [Process]?
• How likely are you to use [Process] moving forward?
• Please rate your level of agreement with the following statement: I find a lot of value in [Process].

While these are great examples of what evaluation research questions to ask, keep in mind they should be reflective of your unique goals and objectives. 

Our evaluation research company can help design, program, field, and analyze your survey to assure you are using quality data to drive decision-making.

Contact Our Evaluation Research Company

Wondering if continuing an employee or customer program is still offering value to your organization? Or, perhaps you need to determine if a new product concept is working as effectively as it should be. Evaluation research can help achieve these objectives and plenty of others. 

Drive Research is a full-service market research company specializing in evaluation research through surveys, focus groups, and IDIs.  Contact our team by filling out the form below or emailing [email protected] .

Emily Taylor

As a Research Manager, Emily is approaching a decade of experience in the market research industry and loves to challenge the status quo. She is a certified VoC professional with a passion for storytelling.

Learn more about Emily, here .

Categories: Market Research Glossary

Need help with your project? Get in touch with Drive Research.

View Our Blog

Program Evaluation Research

Cite this chapter.

• Jerald Jay Thomas  

1469 Accesses

According to the Joint Committee on Standards for Educational Evaluation (2011), program evaluation as a method of research is a means of systematically evaluating an object or educational program. As straightforward and succinct as that definition is, you will find that evaluation research borrows heavily from other methods of research. Evaluation research has at its root the assumption that the value, quality, and effectiveness of an educational program can be appraised through a variety of data sources. As educators, we find ourselves making evaluations daily, and in a variety of contexts. The evaluations we make according to Fitzpatrick, Sanders, and Worthen (2011) fall along a continuum between formal evaluation and informal evaluation.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save.

• Get 10 units per month
• Download Article/Chapter or eBook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime
• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info
• Durable hardcover edition

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Unable to display preview.  Download preview PDF.

Similar content being viewed by others

Evaluating Educational Programs

Evaluation Paradigms

A Case Study for the 10-Step Approach to Program Evaluation

American Educational Research Association. (2011). Codeof ethics of the American Educational Research Association . Retrieved from http://www.aera.net/uploadedFiles/About_AERA/Ethical_Standards/CodeOfEthics%281%29.pdf.

Google Scholar  

Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2011). Program evaluation: Alternative approaches and practical guidelines . Upper Saddle River, NJ: Pearson Education.

Greene, J. C. (2005). Mixed methods. In S. Mathison (Ed.), Encyclopedia of evaluation (pp. 397–298). Thousand Oaks, CA: Sage.

Joint Committee on Standards for Educational Evaluation. (2011). The program evaluation standards . Newbury Park, CA: Sage.

McNeil, K., Newman, I., & Steinhauser, J. (2005). How to be involved in program evaluation . Lanham, MD: Scarecrow Education.

Sanders, J. R. (2000). Evaluating school programs: An educator’s guide. Thousand Oaks, CA: Corwin Press.

Scriven, M. (1991). Beyond formative and summative evaluation. In G. W. McLaughlin & D. C. Phillips (Eds.), Evaluation and education: A quarter century (pp. 19–64). Chicago, IL: University of Chicago Press.

Download references

You can also search for this author in PubMed   Google Scholar

Editor information

Copyright information.

© 2012 Sheri R. Klein

About this chapter

Thomas, J.J. (2012). Program Evaluation Research. In: Klein, S.R. (eds) Action Research Methods. Palgrave Macmillan, New York. https://doi.org/10.1057/9781137046635_9

Download citation

DOI : https://doi.org/10.1057/9781137046635_9

Publisher Name : Palgrave Macmillan, New York

Print ISBN : 978-1-349-29560-9

Online ISBN : 978-1-137-04663-5

eBook Packages : Palgrave Education Collection Education (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

• Search Menu

Sign in through your institution

• Browse content in Arts and Humanities
• Browse content in Archaeology
• Anglo-Saxon and Medieval Archaeology
• Archaeological Methodology and Techniques
• Archaeology by Region
• Archaeology of Religion
• Archaeology of Trade and Exchange
• Biblical Archaeology
• Contemporary and Public Archaeology
• Environmental Archaeology
• Historical Archaeology
• History and Theory of Archaeology
• Industrial Archaeology
• Landscape Archaeology
• Mortuary Archaeology
• Prehistoric Archaeology
• Underwater Archaeology
• Zooarchaeology
• Browse content in Architecture
• Architectural Structure and Design
• History of Architecture
• Residential and Domestic Buildings
• Theory of Architecture
• Browse content in Art
• Art Subjects and Themes
• History of Art
• Industrial and Commercial Art
• Theory of Art
• Biographical Studies
• Byzantine Studies
• Browse content in Classical Studies
• Classical History
• Classical Philosophy
• Classical Mythology
• Classical Numismatics
• Classical Literature
• Classical Reception
• Classical Art and Architecture
• Classical Oratory and Rhetoric
• Greek and Roman Epigraphy
• Greek and Roman Law
• Greek and Roman Archaeology
• Greek and Roman Papyrology
• Late Antiquity
• Religion in the Ancient World
• Social History
• Digital Humanities
• Browse content in History
• Colonialism and Imperialism
• Diplomatic History
• Environmental History
• Genealogy, Heraldry, Names, and Honours
• Genocide and Ethnic Cleansing
• Historical Geography
• History by Period
• History of Agriculture
• History of Education
• History of Emotions
• History of Gender and Sexuality
• Industrial History
• Intellectual History
• International History
• Labour History
• Legal and Constitutional History
• Local and Family History
• Maritime History
• Military History
• National Liberation and Post-Colonialism
• Oral History
• Political History
• Public History
• Regional and National History
• Revolutions and Rebellions
• Slavery and Abolition of Slavery
• Social and Cultural History
• Theory, Methods, and Historiography
• Urban History
• World History
• Browse content in Language Teaching and Learning
• Language Learning (Specific Skills)
• Language Teaching Theory and Methods
• Browse content in Linguistics
• Applied Linguistics
• Cognitive Linguistics
• Computational Linguistics
• Forensic Linguistics
• Grammar, Syntax and Morphology
• Historical and Diachronic Linguistics
• History of English
• Language Acquisition
• Language Variation
• Language Families
• Language Evolution
• Language Reference
• Lexicography
• Linguistic Theories
• Linguistic Typology
• Linguistic Anthropology
• Phonetics and Phonology
• Psycholinguistics
• Sociolinguistics
• Translation and Interpretation
• Writing Systems
• Browse content in Literature
• Bibliography
• Children's Literature Studies
• Literary Studies (Asian)
• Literary Studies (European)
• Literary Studies (Eco-criticism)
• Literary Studies (Modernism)
• Literary Studies (Romanticism)
• Literary Studies (American)
• Literary Studies - World
• Literary Studies (1500 to 1800)
• Literary Studies (19th Century)
• Literary Studies (20th Century onwards)
• Literary Studies (African American Literature)
• Literary Studies (British and Irish)
• Literary Studies (Early and Medieval)
• Literary Studies (Fiction, Novelists, and Prose Writers)
• Literary Studies (Gender Studies)
• Literary Studies (Graphic Novels)
• Literary Studies (History of the Book)
• Literary Studies (Plays and Playwrights)
• Literary Studies (Poetry and Poets)
• Literary Studies (Postcolonial Literature)
• Literary Studies (Queer Studies)
• Literary Studies (Science Fiction)
• Literary Studies (Travel Literature)
• Literary Studies (War Literature)
• Literary Studies (Women's Writing)
• Literary Theory and Cultural Studies
• Mythology and Folklore
• Shakespeare Studies and Criticism
• Browse content in Media Studies
• Browse content in Music
• Applied Music
• Dance and Music
• Ethics in Music
• Ethnomusicology
• Gender and Sexuality in Music
• Medicine and Music
• Music Cultures
• Music and Religion
• Music and Culture
• Music and Media
• Music Education and Pedagogy
• Music Theory and Analysis
• Musical Scores, Lyrics, and Libretti
• Musical Structures, Styles, and Techniques
• Musicology and Music History
• Performance Practice and Studies
• Race and Ethnicity in Music
• Sound Studies
• Browse content in Performing Arts
• Browse content in Philosophy
• Aesthetics and Philosophy of Art
• Epistemology
• Feminist Philosophy
• History of Western Philosophy
• Meta-Philosophy
• Metaphysics
• Moral Philosophy
• Non-Western Philosophy
• Philosophy of Science
• Philosophy of Action
• Philosophy of Law
• Philosophy of Religion
• Philosophy of Language
• Philosophy of Mind
• Philosophy of Perception
• Philosophy of Mathematics and Logic
• Practical Ethics
• Social and Political Philosophy
• Browse content in Religion
• Biblical Studies
• Christianity
• East Asian Religions
• History of Religion
• Judaism and Jewish Studies
• Qumran Studies
• Religion and Education
• Religion and Health
• Religion and Politics
• Religion and Science
• Religion and Law
• Religion and Art, Literature, and Music
• Religious Studies
• Browse content in Society and Culture
• Cookery, Food, and Drink
• Cultural Studies
• Customs and Traditions
• Ethical Issues and Debates
• Hobbies, Games, Arts and Crafts
• Natural world, Country Life, and Pets
• Popular Beliefs and Controversial Knowledge
• Sports and Outdoor Recreation
• Technology and Society
• Travel and Holiday
• Visual Culture
• Browse content in Law
• Arbitration
• Browse content in Company and Commercial Law
• Commercial Law
• Company Law
• Browse content in Comparative Law
• Systems of Law
• Competition Law
• Browse content in Constitutional and Administrative Law
• Government Powers
• Judicial Review
• Local Government Law
• Military and Defence Law
• Parliamentary and Legislative Practice
• Construction Law
• Contract Law
• Browse content in Criminal Law
• Criminal Procedure
• Criminal Evidence Law
• Sentencing and Punishment
• Employment and Labour Law
• Environment and Energy Law
• Browse content in Financial Law
• Banking Law
• Insolvency Law
• History of Law
• Human Rights and Immigration
• Intellectual Property Law
• Browse content in International Law
• Private International Law and Conflict of Laws
• Public International Law
• IT and Communications Law
• Jurisprudence and Philosophy of Law
• Law and Politics
• Law and Society
• Browse content in Legal System and Practice
• Courts and Procedure
• Legal Skills and Practice
• Legal System - Costs and Funding
• Primary Sources of Law
• Regulation of Legal Profession
• Medical and Healthcare Law
• Browse content in Policing
• Criminal Investigation and Detection
• Police and Security Services
• Police Procedure and Law
• Police Regional Planning
• Browse content in Property Law
• Personal Property Law
• Restitution
• Study and Revision
• Terrorism and National Security Law
• Browse content in Trusts Law
• Wills and Probate or Succession
• Browse content in Medicine and Health
• Browse content in Allied Health Professions
• Arts Therapies
• Clinical Science
• Dietetics and Nutrition
• Occupational Therapy
• Operating Department Practice
• Physiotherapy
• Radiography
• Speech and Language Therapy
• Browse content in Anaesthetics
• General Anaesthesia
• Browse content in Clinical Medicine
• Acute Medicine
• Cardiovascular Medicine
• Clinical Genetics
• Clinical Pharmacology and Therapeutics
• Dermatology
• Endocrinology and Diabetes
• Gastroenterology
• Genito-urinary Medicine
• Geriatric Medicine
• Infectious Diseases
• Medical Oncology
• Medical Toxicology
• Pain Medicine
• Palliative Medicine
• Rehabilitation Medicine
• Respiratory Medicine and Pulmonology
• Rheumatology
• Sleep Medicine
• Sports and Exercise Medicine
• Clinical Neuroscience
• Community Medical Services
• Critical Care
• Emergency Medicine
• Forensic Medicine
• Haematology
• History of Medicine
• Browse content in Medical Dentistry
• Oral and Maxillofacial Surgery
• Paediatric Dentistry
• Restorative Dentistry and Orthodontics
• Surgical Dentistry
• Medical Ethics
• Browse content in Medical Skills
• Clinical Skills
• Communication Skills
• Nursing Skills
• Surgical Skills
• Medical Statistics and Methodology
• Browse content in Neurology
• Clinical Neurophysiology
• Neuropathology
• Nursing Studies
• Browse content in Obstetrics and Gynaecology
• Gynaecology
• Occupational Medicine
• Ophthalmology
• Otolaryngology (ENT)
• Browse content in Paediatrics
• Neonatology
• Browse content in Pathology
• Chemical Pathology
• Clinical Cytogenetics and Molecular Genetics
• Histopathology
• Medical Microbiology and Virology
• Patient Education and Information
• Browse content in Pharmacology
• Psychopharmacology
• Browse content in Popular Health
• Caring for Others
• Complementary and Alternative Medicine
• Self-help and Personal Development
• Browse content in Preclinical Medicine
• Cell Biology
• Molecular Biology and Genetics
• Reproduction, Growth and Development
• Primary Care
• Professional Development in Medicine
• Browse content in Psychiatry
• Addiction Medicine
• Child and Adolescent Psychiatry
• Forensic Psychiatry
• Learning Disabilities
• Old Age Psychiatry
• Psychotherapy
• Browse content in Public Health and Epidemiology
• Epidemiology
• Public Health
• Browse content in Radiology
• Clinical Radiology
• Interventional Radiology
• Nuclear Medicine
• Radiation Oncology
• Reproductive Medicine
• Browse content in Surgery
• Cardiothoracic Surgery
• Gastro-intestinal and Colorectal Surgery
• General Surgery
• Neurosurgery
• Paediatric Surgery
• Peri-operative Care
• Plastic and Reconstructive Surgery
• Surgical Oncology
• Transplant Surgery
• Trauma and Orthopaedic Surgery
• Vascular Surgery
• Browse content in Science and Mathematics
• Browse content in Biological Sciences
• Aquatic Biology
• Biochemistry
• Bioinformatics and Computational Biology
• Developmental Biology
• Ecology and Conservation
• Evolutionary Biology
• Genetics and Genomics
• Microbiology
• Molecular and Cell Biology
• Natural History
• Plant Sciences and Forestry
• Research Methods in Life Sciences
• Structural Biology
• Systems Biology
• Zoology and Animal Sciences
• Browse content in Chemistry
• Analytical Chemistry
• Computational Chemistry
• Crystallography
• Environmental Chemistry
• Industrial Chemistry
• Inorganic Chemistry
• Materials Chemistry
• Medicinal Chemistry
• Mineralogy and Gems
• Organic Chemistry
• Physical Chemistry
• Polymer Chemistry
• Study and Communication Skills in Chemistry
• Theoretical Chemistry
• Browse content in Computer Science
• Artificial Intelligence
• Computer Architecture and Logic Design
• Game Studies
• Human-Computer Interaction
• Mathematical Theory of Computation
• Programming Languages
• Software Engineering
• Systems Analysis and Design
• Virtual Reality
• Browse content in Computing
• Business Applications
• Computer Security
• Computer Games
• Computer Networking and Communications
• Digital Lifestyle
• Graphical and Digital Media Applications
• Operating Systems
• Browse content in Earth Sciences and Geography
• Atmospheric Sciences
• Environmental Geography
• Geology and the Lithosphere
• Maps and Map-making
• Meteorology and Climatology
• Oceanography and Hydrology
• Palaeontology
• Physical Geography and Topography
• Regional Geography
• Soil Science
• Urban Geography
• Browse content in Engineering and Technology
• Agriculture and Farming
• Biological Engineering
• Civil Engineering, Surveying, and Building
• Electronics and Communications Engineering
• Energy Technology
• Engineering (General)
• Environmental Science, Engineering, and Technology
• History of Engineering and Technology
• Mechanical Engineering and Materials
• Technology of Industrial Chemistry
• Transport Technology and Trades
• Browse content in Environmental Science
• Applied Ecology (Environmental Science)
• Conservation of the Environment (Environmental Science)
• Environmental Sustainability
• Environmentalist Thought and Ideology (Environmental Science)
• Management of Land and Natural Resources (Environmental Science)
• Natural Disasters (Environmental Science)
• Nuclear Issues (Environmental Science)
• Pollution and Threats to the Environment (Environmental Science)
• Social Impact of Environmental Issues (Environmental Science)
• History of Science and Technology
• Browse content in Materials Science
• Ceramics and Glasses
• Composite Materials
• Metals, Alloying, and Corrosion
• Nanotechnology
• Browse content in Mathematics
• Applied Mathematics
• Biomathematics and Statistics
• History of Mathematics
• Mathematical Education
• Mathematical Finance
• Mathematical Analysis
• Numerical and Computational Mathematics
• Probability and Statistics
• Pure Mathematics
• Browse content in Neuroscience
• Cognition and Behavioural Neuroscience
• Development of the Nervous System
• Disorders of the Nervous System
• History of Neuroscience
• Invertebrate Neurobiology
• Molecular and Cellular Systems
• Neuroendocrinology and Autonomic Nervous System
• Neuroscientific Techniques
• Sensory and Motor Systems
• Browse content in Physics
• Astronomy and Astrophysics
• Atomic, Molecular, and Optical Physics
• Biological and Medical Physics
• Classical Mechanics
• Computational Physics
• Condensed Matter Physics
• Electromagnetism, Optics, and Acoustics
• History of Physics
• Mathematical and Statistical Physics
• Measurement Science
• Nuclear Physics
• Particles and Fields
• Plasma Physics
• Quantum Physics
• Relativity and Gravitation
• Semiconductor and Mesoscopic Physics
• Browse content in Psychology
• Affective Sciences
• Clinical Psychology
• Cognitive Neuroscience
• Cognitive Psychology
• Criminal and Forensic Psychology
• Developmental Psychology
• Educational Psychology
• Evolutionary Psychology
• Health Psychology
• History and Systems in Psychology
• Music Psychology
• Neuropsychology
• Organizational Psychology
• Psychological Assessment and Testing
• Psychology of Human-Technology Interaction
• Psychology Professional Development and Training
• Research Methods in Psychology
• Social Psychology
• Browse content in Social Sciences
• Browse content in Anthropology
• Anthropology of Religion
• Human Evolution
• Medical Anthropology
• Physical Anthropology
• Regional Anthropology
• Social and Cultural Anthropology
• Theory and Practice of Anthropology
• Browse content in Business and Management
• Business Strategy
• Business History
• Business Ethics
• Business and Government
• Business and Technology
• Business and the Environment
• Comparative Management
• Corporate Governance
• Corporate Social Responsibility
• Entrepreneurship
• Health Management
• Human Resource Management
• Industrial and Employment Relations
• Industry Studies
• Information and Communication Technologies
• International Business
• Knowledge Management
• Management and Management Techniques
• Operations Management
• Organizational Theory and Behaviour
• Pensions and Pension Management
• Public and Nonprofit Management
• Social Issues in Business and Management
• Strategic Management
• Supply Chain Management
• Browse content in Criminology and Criminal Justice
• Criminal Justice
• Criminology
• Forms of Crime
• International and Comparative Criminology
• Youth Violence and Juvenile Justice
• Development Studies
• Browse content in Economics
• Agricultural, Environmental, and Natural Resource Economics
• Asian Economics
• Behavioural Finance
• Behavioural Economics and Neuroeconomics
• Econometrics and Mathematical Economics
• Economic Systems
• Economic Methodology
• Economic History
• Economic Development and Growth
• Financial Markets
• Financial Institutions and Services
• General Economics and Teaching
• Health, Education, and Welfare
• History of Economic Thought
• International Economics
• Labour and Demographic Economics
• Law and Economics
• Macroeconomics and Monetary Economics
• Microeconomics
• Public Economics
• Urban, Rural, and Regional Economics
• Welfare Economics
• Browse content in Education
• Adult Education and Continuous Learning
• Care and Counselling of Students
• Early Childhood and Elementary Education
• Educational Equipment and Technology
• Educational Strategies and Policy
• Higher and Further Education
• Organization and Management of Education
• Philosophy and Theory of Education
• Schools Studies
• Secondary Education
• Teaching of a Specific Subject
• Teaching of Specific Groups and Special Educational Needs
• Teaching Skills and Techniques
• Browse content in Environment
• Applied Ecology (Social Science)
• Climate Change
• Conservation of the Environment (Social Science)
• Environmentalist Thought and Ideology (Social Science)
• Management of Land and Natural Resources (Social Science)
• Natural Disasters (Environment)
• Pollution and Threats to the Environment (Social Science)
• Social Impact of Environmental Issues (Social Science)
• Sustainability
• Browse content in Human Geography
• Cultural Geography
• Economic Geography
• Political Geography
• Browse content in Interdisciplinary Studies
• Communication Studies
• Museums, Libraries, and Information Sciences
• Browse content in Politics
• African Politics
• Asian Politics
• Chinese Politics
• Comparative Politics
• Conflict Politics
• Elections and Electoral Studies
• Environmental Politics
• Ethnic Politics
• European Union
• Foreign Policy
• Gender and Politics
• Human Rights and Politics
• Indian Politics
• International Relations
• International Organization (Politics)
• Irish Politics
• Latin American Politics
• Middle Eastern Politics
• Political Methodology
• Political Communication
• Political Philosophy
• Political Sociology
• Political Theory
• Political Behaviour
• Political Economy
• Political Institutions
• Politics and Law
• Politics of Development
• Public Administration
• Public Policy
• Qualitative Political Methodology
• Quantitative Political Methodology
• Regional Political Studies
• Russian Politics
• Security Studies
• State and Local Government
• UK Politics
• US Politics
• Browse content in Regional and Area Studies
• African Studies
• Asian Studies
• East Asian Studies
• Japanese Studies
• Latin American Studies
• Middle Eastern Studies
• Native American Studies
• Scottish Studies
• Browse content in Research and Information
• Research Methods
• Browse content in Social Work
• Addictions and Substance Misuse
• Adoption and Fostering
• Care of the Elderly
• Child and Adolescent Social Work
• Couple and Family Social Work
• Direct Practice and Clinical Social Work
• Emergency Services
• Human Behaviour and the Social Environment
• International and Global Issues in Social Work
• Mental and Behavioural Health
• Social Justice and Human Rights
• Social Policy and Advocacy
• Social Work and Crime and Justice
• Social Work Macro Practice
• Social Work Practice Settings
• Social Work Research and Evidence-based Practice
• Welfare and Benefit Systems
• Browse content in Sociology
• Childhood Studies
• Community Development
• Comparative and Historical Sociology
• Disability Studies
• Economic Sociology
• Gender and Sexuality
• Gerontology and Ageing
• Health, Illness, and Medicine
• Marriage and the Family
• Migration Studies
• Occupations, Professions, and Work
• Organizations
• Population and Demography
• Race and Ethnicity
• Social Theory
• Social Movements and Social Change
• Social Research and Statistics
• Social Stratification, Inequality, and Mobility
• Sociology of Religion
• Sociology of Education
• Sport and Leisure
• Urban and Rural Studies
• Browse content in Warfare and Defence
• Defence Strategy, Planning, and Research
• Land Forces and Warfare
• Military Administration
• Military Life and Institutions
• Naval Forces and Warfare
• Other Warfare and Defence Issues
• Peace Studies and Conflict Resolution
• Weapons and Equipment

• < Previous chapter
• Next chapter >

17 Program Evaluation: Principles, Procedures, and Practices

Aurelio José Figueredo, Department of Psychology, School of Mind, Brain, and Behavior, Division of Family Studies and Human Development, College of Agriculture and Life Sciences, University of Arizona, Tucson, AZ

Sally Gayle Olderbak, Department of Psychology, School of Mind, Brain, and Behavior, Division of Family Studies and Human Development, College of Agriculture and Life Sciences, University of Arizona, Tucson, AZ

Gabriel Lee Schlomer, Division of Family Studies and Human Development, College of Agriculture and Life Sciences, University of Arizona, Tucson, AZ

Rafael Antonio Garcia, Division of Family Studies and Human Development, College of Agriculture and Life Sciences, University of Arizona, Tucson, AZ

Pedro Sofio Abril Wolf, Department of Psychology, University of Cape Town, South Africa

• Published: 01 October 2013
• Cite Icon Cite
• Permissions Icon Permissions

This chapter provides a review of the current state of the principles, procedures, and practices within program evaluation. We address a few incisive and difficult questions about the current state of the field: (1) What are the kinds of program evaluations? (2) Why do program evaluation results often have so little impact on social policy? (3) Does program evaluation suffer from a counterproductive system of incentives? and (4) What do program evaluators actually do? We compare and contrast the merits and limitations, strengths and weaknesses, and relative progress of the two primary contemporary movements within program evaluation, Quantitative Methods and Qualitative Methods, and we propose an epistemological framework for integrating the two movements as complementary forms of investigation, each contributing to different stages in the scientific process. In the final section, we provide recommendations for systemic institutional reforms addressing identified structural problems within the real-world practice of program evaluation.

Personal account

• Sign in with email/username & password
• Get email alerts
• Save searches
• Purchase content
• Activate your purchase/trial code
• Add your ORCID iD

Institutional access

Sign in with a library card.

• Sign in with username/password
• Recommend to your librarian
• Institutional account management
• Get help with access

Access to content on Oxford Academic is often provided through institutional subscriptions and purchases. If you are a member of an institution with an active account, you may be able to access content in one of the following ways:

IP based access

Typically, access is provided across an institutional network to a range of IP addresses. This authentication occurs automatically, and it is not possible to sign out of an IP authenticated account.

Choose this option to get remote access when outside your institution. Shibboleth/Open Athens technology is used to provide single sign-on between your institution’s website and Oxford Academic.

• Click Sign in through your institution.
• Select your institution from the list provided, which will take you to your institution's website to sign in.
• When on the institution site, please use the credentials provided by your institution. Do not use an Oxford Academic personal account.
• Following successful sign in, you will be returned to Oxford Academic.

If your institution is not listed or you cannot sign in to your institution’s website, please contact your librarian or administrator.

Enter your library card number to sign in. If you cannot sign in, please contact your librarian.

Society Members

Society member access to a journal is achieved in one of the following ways:

Sign in through society site

Many societies offer single sign-on between the society website and Oxford Academic. If you see ‘Sign in through society site’ in the sign in pane within a journal:

• Click Sign in through society site.
• When on the society site, please use the credentials provided by that society. Do not use an Oxford Academic personal account.

If you do not have a society account or have forgotten your username or password, please contact your society.

Sign in using a personal account

Some societies use Oxford Academic personal accounts to provide access to their members. See below.

A personal account can be used to get email alerts, save searches, purchase content, and activate subscriptions.

Some societies use Oxford Academic personal accounts to provide access to their members.

Viewing your signed in accounts

Click the account icon in the top right to:

• View your signed in personal account and access account management features.
• View the institutional accounts that are providing access.

Signed in but can't access content

Oxford Academic is home to a wide variety of products. The institutional subscription may not cover the content that you are trying to access. If you believe you should have access to that content, please contact your librarian.

For librarians and administrators, your personal account also provides access to institutional account management. Here you will find options to view and activate subscriptions, manage institutional settings and access options, access usage statistics, and more.

Our books are available by subscription or purchase to libraries and institutions.

• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Rights and permissions
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

10 23. Program evaluation

Chapter outline.

• What is program evaluation? (5 minute read time)
• Planning your program evaluation (20 minute read time, including video)
• Process evaluations and implementation science (7 minute read time)
• Outcome and impact evaluations (5 minute read time)
• Ethics and culture in program evaluation (10 minute read time)

Content warning: discussions of BMI/weight/obesity, genocide, and residential schools for indigenous children.

Imagine you are working for a nonprofit focused on children’s health and wellness in school. One of the grants you received this year funds a full-time position at a local elementary school for a teacher who will be integrating kinesthetic learning into their lesson plans for math classes for third graders. Kinesthetic learning is learning that occurs when the students do something physical to help learn and reinforce information, instead of listening to a lecture or other verbal teaching activity. You have read research suggesting that students retain information better using kinesthetic teaching methods and that it can reduce student behavior issues. You want to know if it might benefit your community.

When you applied for the grant, you had to come up with some outcome measures that would tell the foundation if your program was worth continuing to fund – if it’s having an effect on your target population (the kids at the school). You told the foundation you would look at three outcomes:

• How did using kinesthetic learning affect student behavior in classes?
• How did using kinesthetic learning affect student scores on end-of-year standardized tests?
• How did the students feel about kinesthetic teaching methods?

But, you say, this sounds like research! However, we have to take a look at the purpose, origin, effect , and execution of the project to understand the difference, which we do in section 23.1 in this chapter. Those domains are where we can find the similarities and differences between program evaluation and research.

Realistically, as a practitioner, you’re far more likely to engage in program evaluation than you are in research. So, you might ask why you are learning research methods and not program evaluation methods, and the answer is that you will use research methods in evaluating programs. Program evaluation tends to focus less on generalizability, experimental design, and replicability, and instead focuses on the practical application of research methods to a specific context in practice.

23.1 What is program evaluation?

Learning objectives.

Learners will be able to…

• Define program evaluation
• Discuss similarities and differences between program evaluation and research
• Determine situations in which program evaluation is more appropriate than research

Program evaluation can be defined as the systematic process by which we determine if social programs are meeting their goals, how well the program runs, whether the program had the desired effect, and whether the program has merit according to stakeholders (including in terms of the monetary costs and benefits). It’s important to know what we mean when we say “evaluation.” Pruett (2000) [1]  provides a useful definition: “Evaluation is the systematic application of scientific methods to assess the design, implementation, improvement or outcomes of a program” (para. 1). That nod to scientific methods is what ties program evaluation back to research, as we discussed above. Program evaluation is action-oriented, which makes it fit well into social work research (as we discussed in Chapter 1 ).

Often, program evaluation will consist of mixed methods because its focus of is so heavily on the effect of the program in your specific context . Not that research doesn’t care about the effects of programs – of course it does! But with program evaluation, we seek to ensure the way that we are applying our program works in our agency, with our communities and clients. Thinking back to the example at the beginning of the chapter, consider the following: Does kinesthetic learning make sense for your school? What if your classroom spaces are too small? Are the activities appropriate for children with differing physical abilities who attend your school? What if school administrators are on board, but some parents are skeptical?

The project we talked about in the introductions – a real project, by the way – was funded by a grant from a foundation. The reality of the grant funding environment is that funders want to see that their money is not only being used wisely, but is having a material effect on the target population. This is a good thing, because we want to know our programs have a positive effect on clients and communities. We don’t want to just keep running a program because it’s what we’ve always done. (Consider the ethical implications of continuing to run an ineffective program.) It also forces us as practitioners to plan grant-funded programs with an eye toward evaluation. It’s much easier to evaluate your program when you can gather data at the beginning of the program than when you have to work backwards at the middle or end of the program.

How do program evaluation and research relate to each other?

As we talked about above, program evaluation and research are similar, particularly in that they both rely on scientific methods. Both use quantitative and qualitative methods, like data analysis and interviews. Effective program evaluation necessarily involves the research methods we’ve talked about in this book. Without understanding research methods, your program evaluation won’t be very rigorous and probably won’t give you much useful information.

However, there are some key differences between the two that render them distinct activities that are appropriate in different circumstances. Research is often exploratory and not evaluative at all, and instead looks for relationships between variables to build knowledge on a subject. It’s important to note at the outset that what we’re discussing below is not universally true of all projects. Instead, the framework we’re providing is a broad way to think about the differences between program evaluation and research. Scholars and practitioners disagree on whether program evaluation is a subset of research or something else entirely (and everything in between). The important thing to know about that debate is that it’s not settled, and what we’re presenting below is just one way to think about the relationship between the two.

According to Mathison (2008) [2] , the differences between program evaluation and research have to do with the domains of purpose, origins, effect and execution. 

Let’s think back to our example from the start of the chapter – kinesthetic teaching methods for 3rd grade math – to talk more about these four domains.

To understand this domain, we have to ask a few questions: why do we want to research or evaluate this program? What do we hope to gain? This is the  why  of our project (Mathison). Another way to think about it is as the  aim  of your research, which is a concept you hopefully remember from Chapter 2.

Through the lens of program evaluation, we’re evaluating this program because we want to know its effects, but also because our funder probably only wants to give money to programs that do what they’re supposed to do. We want to gather information to determine if it’s worth it for our funder – or for  us  – to invest resources in the program.

If this were a research project instead, our purpose would be congruent, but different. We would be seeking to add to the body of knowledge and evidence about kinesthetic learning, most likely hoping to provide information that can be generalized beyond 3rd grade math students. We’re trying to inform further development of the body of knowledge around kinesthetic learning and children. We’d also like to know if and how we can apply this program in contexts other than one specific school’s 3rd grade math classes. These are not the only research considerations, but just a few examples.

Purpose and origins can feel very similar and be a little hard to distinguish. The main difference is that origins are about the  who , whereas purpose is about the  why  (Mathison). So, to understand this domain, we have to ask about the source of our project – who wanted to get the project started? What do they hope this project will contribute?

For a program evaluation, the project usually arises from the priorities of funders, agencies, practitioners and (hopefully) consumers of our services. They are the ones who define the purpose we discussed above and the questions we will ask.

In research, the project arises from a researcher’s intellectual curiosity and desire to add to a body of knowledge around something they think is important and interesting. Researchers define the purpose and the questions asked in the project.

The effect of program evaluation and research is essentially what we’re going to use our results for. For program evaluation, we will use them to make a decision about whether a program is worth continuing, what changes we might make to the program in the future or how we might change the resources we devote going forward. The results are often also used by our funders to make decisions about whether they want to keep funding our program or not. (Outcome evaluations aren’t the only thing that funders will look at – they also sometimes want to know whether our processes in the program were faithful to what we described when we requested funding. We’ll discuss outcome and process evaluations in section 23.4.)

The effect of research – again, what we’re going to use our results for – is typically to add to the knowledge and evidence base surrounding our topic. Research can certainly be used for decision-making about programs, especially to decide which program to implement in the first place. But that’s not what results are primarily used for, especially by other researchers.

Execution is fundamentally the  how  of our project. What are the circumstances under which we’re running the project?

Program evaluation projects that most of us will ever work on are frequently based in a nonprofit or government agency. Context is extremely important in program evaluation (and program implementation). As most of us will know, these are environments with lots of moving parts. As a result, running controlled experiments is usually not possible, and we sometimes have to be more flexible with our evaluations to work with the resources we actually have and the unique challenges and needs of our agencies. This doesn’t mean that program evaluations can’t be rigorous or use strong research methods. We just have to be realistic about our environments and plan for that when we’re planning our evaluation.

Research is typically a lot more controlled. We do everything we can to minimize outside influences on our variables of interest, which is expected of rigorous research. Of course, some research is  extremely  controlled, especially experimental research and randomized controlled trials. this all ties back to the purpose, origins, and effects of research versus those of program evaluation – we’re primarily building knowledge and evidence.

In the end, it’s important to remember that these are guidelines, and you will no doubt encounter program evaluation projects that cross the lines of research, and vice versa. Understanding how the two differ will help you decide how to move forward when you encounter the need to assess the effect of a program in practice.

Key Takeaways

• Program evaluation is a systematic process that uses the scientific research method to determine the effects of social programs.
• Program evaluation and research are similar, but they differ in purpose, origins, effect and execution.
• The purpose of program evaluation is to judge the merit or worth of a program, whereas the purpose of research is primarily to contribute to the body of knowledge around a topic.
• The origins of program evaluation are usually funders and people working in agencies, whereas research originates primarily with scholars and their scientific interests.
• Program evaluations are typically used to make decisions about programs, whereas research is used to add to the knowledge and evidence base around a topic.
• Executing a program evaluation project requires a strong understanding of your setting and context in order to adapt your evaluation to meet your goals in a realistic way. The execution of research is much more controlled and seeks to minimize the influence of context.
• If you were conducting a research project on the kinesthetic teaching methods that we talked about in this chapter, what is one research question you could study that aligns with the purpose, origins, and effects of research?
• Consider the research project you’ve been building throughout this book. What is one program evaluation question you could study that aligns with the purpose, origins, and effects of program evaluation? How might its execution look different than what you’ve envisioned so far?

23.2 Planning your program evaluation

• Discuss how planning a program evaluation is similar and different from planning a research project
• Identify program stakeholders
• Identify the basics of logic models and how they inform evaluation
• Produce evaluation questions based on a logic model

Planning a program evaluation project requires just as much care and thought as planning a research project. But as we discussed in section 23.1, there are some significant differences between program evaluation and research that mean your planning process is also going to look a little different. You have to involve the program stakeholders at a greater level than that found with most types of research, which will sometimes focus your program evaluation project on areas you wouldn’t have necessarily chosen (for better or worse). Your program evaluation questions are far less likely to be exploratory; they are typically evaluative and sometimes explanatory.

For instance, I worked on a project designed to increase physical activity for elementary school students at recess. The school had noticed a lot of kids would just sit around at recess instead of playing. As an intervention, the organization I was working with hired recess coaches to engage the kids with new games and activities to get them moving. Our plan to measure the effect of recess coaching was to give the kids pedometers at a couple of different points during the year, and see if there was any change in their activity level as measured by the number of steps they took during recess. However, the school was also concerned with the rate of obesity among students, and asked us to also measure the height and weight of the students to calculate BMI at the beginning and end of the year. I balked at this because kids are still growing and BMI isn’t a great measure to use for kids and some kids were uncomfortable with us weighing them (with parental consent), even though no other kids would be in the room. However, the school was insistent that we take those measurements, and so we did that for all kids whose parents consented and who themselves assented to have their weight measured. We didn’t think BMI was an important measure, but the school did, so this changed an element of our evaluation.

In an ideal world, your program evaluation is going to be part of your overall program plan. This very often doesn’t happen in practice, but for the purposes of this section, we’re going to assume you’re starting from scratch with a program and really internalized the first sentence of this paragraph. (It’s important to note that no one  intentionally leaves evaluation out of their program planning; instead, it’s just not something many people running programs think about. They’re too busy… well, running programs. That’s why this chapter is so important!)

In this section, we’re going to learn about how to plan your program evaluation, including the importance of logic models. You may have heard people groan about logic models (or you may have groaned when you read those words), and the truth is, they’re a lot of work and a little complicated. Teaching you how to make one from start to finish is a little bit outside the scope of this section, but what I am going to try to do is teach you how to interpret them and build some evaluation questions from them. (Pro-tip: logic models are a heck of a lot easier to make in Excel than Word.)

It has three primary steps: engaging stakeholders, describing the program and focusing the evaluation.

Step 1: Engaging stakeholders

Stakeholders are the people and organizations that have some interest in or will be impacted by our program. Including as many stakeholders as possible when you plan your evaluation will help to make it as useful as possible for as many people as possible. The key to this step is to listen. However, a note of caution: sometimes stakeholders have competing priorities, and as the program evaluator, you’re going to have to help navigate that. For example, in our kinesthetic learning program, the teachers at your school might be interested in decreasing classroom disruptions or enhancing subject matter learning, while the administration is solely focused on test scores, while the administration is solely focused on test scores. Here is where it’s a great idea to use your social work ethics and research knowledge to guide conversations and planning. Improved test scores are great, but how much does that actually  benefit the students?

Step 2: Describe the program

Once you’ve got stakeholder input on evaluation priorities, it’s time to describe what’s going into the program and what you hope your participants and stakeholders will get out of it. Here is where a logic model becomes an essential piece of program evaluation. A logic model “ is a graphic depiction (road map) that presents the shared relationships among the resources, activities, outputs, outcomes, and impact for your program” ( Centers for Disease Control , 2018, para. 1). Basically, it’s a way to show how what you’re doing is going to lead to an intended outcome and/or impact. (We’ll discuss the difference between outcomes and impacts in section 23.4.)

Logic models have several key components, which I describe in the list below (CDC, 2018). The components are numbered because of where they come in the “logic” of your program – basically, where they come in time order.

• Inputs: resources (e.g. people and material resources) that you have to execute your program.
• Activities: what you’re actually doing with your program resources.
• Outputs: the direct products and results of your program.
• Outcomes: the changes that happen because of your program inputs and activities.
• Impacts: the long-term effects of your program.

The CDC also talks about moderators – what they call “contextual factors” – that affect the execution of your program evaluation. This is an important component of the execution of your project, which we talked about in 23.1. Context will also become important when we talk about implementation science in section 23.3.

Let’s think about our kinesthetic learning project. While you obviously don’t have full information about what the project looks like, you’ve got a good enough idea for a little exercise below.

Step 3: Focus the evaluation

So now you know what your stakeholder priorities are and you have described your program. It’s time to figure out what questions you want to ask that will reflect stakeholder priorities and are actually possible given your program inputs, activities and outputs.

Why do inputs, activities and outputs matter for your question?

• Inputs are your resources for the evaluation – do you have to do it with existing staff, or can you hire an expert consultant? Realistically, what you ask is going to be affected by the resources you can dedicate to your evaluation project, just like in a research project.
• Activities are what you can actually evaluate – for instance, what effect does using hopscotch to teach multiplication have?
• And finally, outputs are most likely your indicators of change – student engagement with administrators for behavioral issues, end-of-grade math test scores, for example.
• Program evaluation planning should be rigorous like research planning, but will most likely focus more on stakeholder input and evaluative questions
• The three primary steps in planning a program evaluation project are engaging stakeholders, describing your program, and focusing your evaluation.
• Logic models are a key piece of information in planning program evaluation because they describe how a program is designed to work and what you are investing in it, which are important factors in formulating evaluation questions.
• Who would the key stakeholders be? What is each stakeholder’s interest in the project?
• What are the activities (the action(s) you’re evaluating) and outputs (data/indicators) for your program? Can you turn them into an evaluation question?

23.3 Process evaluation and implementation science

• Define process evaluation
• Explain why process evaluation is important for programs
• Distinguish between process and outcome measures
• Explain the purpose of implementation science and how it relates to program evaluation

Something we often don’t have time for in practice is evaluating how things are going internally with our programs. How’s it going with all the documentation our agency asks us to complete? Is the space we’re using for our group sessions facilitating client engagement? Is the way we communicate with volunteers effective? All of these things can be evaluated using a process evaluation , which is an analysis of how well your program ended up running, and sometimes how well it’s going in real time.  If you have the resources and ability to complete one of these analyses, I highly recommend it – even if it stretches your staff, it will often result in a greater degree of efficiency in the long run. (Evaluation should, at least in part, be about the long game.)

From a research perspective, process evaluations can also help you find irregularities in how you collect data that might be affecting your outcome or impact evaluations. Like other evaluations, ideally, you’re going to plan your process evaluation before you start the project. Take an iterative approach, though, because sometimes you’re going to run into problems you need to analyze in real time.

The RAND corporation is an excellent resource for guidance on program evaluation, and they describe process evaluations this way: “Process evaluations typically track attendance of participants, program adherence, and how well you followed your work plan. They may also involve asking about satisfaction of program participants or about staff’s perception of how well the program was delivered. A process evaluation should be planned before the program begins and should continue while the program is running” (RAND Corporation, 2019, para. 1) [3] .

There are several key data sources for process evaluations (RAND Corporation, 2019) [4] , some of which are listed below.

• Participant data: can help you determine if you are actually reaching the people you intend to.
• Focus groups: how did people experience the program? How could you improve it from the participant perspective?
• Satisfaction surveys: did participants get what they wanted from the program?
• Staff perception data: How did the program go for staff? Were expectations realistic? What did they see in terms of qualitative changes for participants?
• Program adherence monitoring: how well did you follow your program plans?

Using these data sources, you can learn lessons about your program and make any necessary adjustments if you run the program again. It can also give you insights about your staff’s needs (like training, for instance) and enable you to identify gaps in your programs or services.

Implementation science: The basics

A further development of process evaluations, i mplementation science is “the scientific study of methods to promote the systematic uptake of research findings and other evidence-based practices into routine practice, and, hence, to improve the quality and effectiveness of health services.” (Bauer, Damschroder, Hagerdorn, Smith & Kilbourne, 2015) [5]

Put more plainly, implementation science studies how we put evidence-based interventions (EBIs) into practice. It’s e ssentially a form of process evaluation, just at a more macro level. Implementation science is a r elatively new field of study that focuses on how to best put interventions into practice, and it’s i mportant because it helps us analyze on a macro level those factors that might affect our ability to implement a program. Implementation science focuses on the context of program implementation, which has significant implications for program evaluation.

A useful framework for implementation science is the EPIS (Exploration, Preparation, Implementation and Sustainment) framework. It’s not the only one out there, but I like it because to me, it sort of mirrors the linear nature of a logic model.

The EPIS framework was developed by Aarons, Hurlburt and Horwitz (first published 2011). (The linked article is behind a paywall, the abstract is still pretty useful, and if you’re affiliated with a college or university, you can probably get access through your library.) This framework emphasizes the importance of the context in which your program is being implemented – inner, organizational, context and outer, or the political, public policy and social contexts. What’s happening in your organization and in the larger political and social sphere that might affect how your program gets implemented?

There are a few key questions in each phase, according to Aarons, Hurlburt and Horwitz (2011) [6] :

• Exploration phase: what is the problem or issue we want to address? What are our options for programs and interventions? What is the best way to put them into practice? What is the organizational and societal context that we need to consider when choosing our option?
• Preparation: which option do we want to adopt? What resources will we need to put that option into practice? What are our organizational or sociopolitical assets and challenges in putting this option into practice?
• Implementation: what is actually happening now that we’re putting our option into practice? How is the course of things being affected by contexts?
• Sustainment: what can we do to ensure our option remains viable, given competing priorities with funding and public attention?

Implementation is a new and rapidly advancing field, and realistically, it’s beyond what a lot of us are going to be able to evaluate in our agencies at this point. But even taking pieces of it – especially the pieces about the importance of context for our programs and evaluations – is useful. Even if you don’t use it as an evaluative framework, the questions outlined above are good ones to ask when you’re planning your program in the first place.

• A  process evaluation is an analysis of how your program actually ran, and sometimes how it’s running in real time.
• Process evaluations are useful because they can help programs run more efficiently and effectively and reveal agency and program needs.
• The EPIS model is a way to analyze the implementation of a program that emphasizes distinct phases of implementation and the context in which the phases happen.
• The EPIS model is also useful in program planning, as it mirrors the linear process of a logic model .
• Consider your research project or, if you have been able to adapt it, your program evaluation project. What are some inner/organizational context factors that might affect how the program gets implemented and what you can evaluate?
• What are some things you would want to evaluate about your program’s process? What would you gain from that information?

23.4 Outcome and impact evaluations

• Define outcome
• Explain the principles of conducting an outcome evaluation
• Define impact
• Explain the principles of conducting an impact evaluation
• Explain the difference between outcomes and impacts

A lot of us will use “outcome” and “impact” interchangeably, but the truth is, they are different. An o utcome is the final condition that occurs at the end of an intervention or program. It is the short-term effect – for our kinesthetic learning example, perhaps an improvement over last year’s end-of-grade math test scores. An i mpact is the long-term condition that occurs at the end of a defined time period after an intervention. It is the longer-term effect – for our kinesthetic learning example, perhaps better retention of math skills as students advance through school. Because of this distinction, outcome and impact evaluations are going to look a little different.

But first, let’s talk about how these types of evaluations are the same. Outcome and impact evaluations are all about change. As a result, we have to know what circumstance, characteristic or condition we are hoping will change because of our program.  We also need to figure out what we think the causal link between our intervention or program and the change is, especially if we are using a new type of intervention that doesn’t yet have a strong evidence base.

For both of these types of evaluations, you have to consider what type of research design you can actually use in your circumstances – are you coming in when a program is already in progress, so you have no baseline data? Or can you collect baseline data to compare to a post-test? For impact evaluations, how are you going to track participants over time?

The main difference between outcome and impact evaluation is the timing and, consequently, the difficulty and level of investment. You can pretty easily collect outcome data from program participants at the end of the program. But tracking people over time, especially for populations social workers serve, can be extremely difficult. It can also be difficult or impossible to control for whatever happened in your participant’s life between the end of the program and the end of your long-term measurement period.

Impact evaluations require careful planning to determine how your follow-up is going to happen. It’s a good practice to try to keep intermittent contact with participants, even if you aren’t taking a measurement at that time, so that you’re less likely to lose track of them.

• Outcomes are short-term effects that can be measured at the end of a program.
• Outcome evaluations apply research methods to the analysis of change during a program and try to establish a logical link between program participation and the short-term change.
• Impacts are long-term effects that are measured after a period of time has passed since the end of a program.
• Impact evaluations apply research methods to the analysis of change after a defined period of time has passed after the end of a program and try to establish a logical link between program participation and long-term change.
• Is each of the following examples an outcome or an impact? Choose the correct answer.

23.5 Ethics and culture in program evaluation

• Discuss cultural and ethical issues to consider when planning and conducting program evaluation
• Explain the importance of stakeholder and participant involvement to address these issues

In a now decades-old paper, Stake and Mabry (1998) [7] point out, “The theory and practice of evaluation are of little value unless we can count on vigorous ethical behavior by evaluators” (p. 99). I know we always say to use the most recent scholarship available, but this point is as relevant now as it was over 20 years ago. One thing they point out that rings particularly true for me as an experienced program evaluator is the idea that we evaluators are also supposed to be “program advocates” (p. 99). We have to work through competing political and ideological differences from our stakeholders, especially funders, that, while sometimes present in research, are especially salient for program evaluation given its origins.

There’s not a rote answer for these ethical questions, just as there are none for the practice-based ethical dilemmas your instructors hammer home with you in classes. You need to use your research and social work ethics to solve these problems. Ultimately, do your best to focus on rigor while meeting stakeholder needs.

One of the most important ethical issues in program evaluation is the implication of not evaluating your program. Providing an ineffective intervention to people can be extremely harmful. And what happens if our intervention actually causes harm? It’s our duty as social workers to explore these issues and not just keep doing what we’ve always done because it’s expedient or guarantees continued funding. I’ve evaluated programs before that turned out to be ineffective, but were required by state law to be delivered to a certain population. It’s not just potentially harmful to clients; it’s also a waste of precious resources that could be devoted to other, more effective programs.

We’ve talked throughout this book about ethical issues and research. All of that is applicable to program evaluation too. Federal law governing IRB practice does not require that program evaluation go through IRB if it is not seeking to gather generalizable knowledge, so IRB approval isn’t a given for these projects. As a result, you’re even more responsible for ensuring that your project is ethical.

Cultural considerations

Ultimately, social workers should start from a place of humility in the face of cultures or groups of which we are not a part. Cultural considerations in program evaluation look similar to those in research. Something to consider about program evaluation, though: is it your duty to point out potential cultural humility issues as part of your evaluation, even if you’re not asked to? I’d argue that it is.

It is also important we make sure that our definition of success is not oppressive. For example, in Australia, the government undertook a program to remove Aboriginal children from their families and assimilated them into white culture.  The program was viewed as successful, but the measures of success were based on oppressive beliefs and stereotypes. This is why stakeholder input is essential – especially if you’re not a member of the group you’re evaluating, stakeholders are going to be the ones to tell you that you may need to reconsider what “success” means.

Unrau , Gabor, and Grinnell (2007) [8] identified several important factors to consider when designing and executing a culturally sensitive program evaluation. First, evaluators need “a clear understanding of the impact of culture on human and social processes generally and on evaluation processes specifically and… skills in cross-cultural communications to ensure that they can effectively interact with people from diverse backgrounds” (p. 419). These are also essential skills in social work practice that you are hopefully learning in your other classes! We should strive to learn as much as possible about the cultures of our clients when they differ from ours.

The authors also point out that evaluators need to be culturally aware and make sure the way they plan and execute their evaluations isn’t centered on their own ethnic experience and that they aren’t basing their plans on stereotypes about other cultures. In addition, when executing our evaluations, we have to be mindful of how our cultural background affects our communication and behavior, because we may need to adjust these to communicate (both verbally and non-verbally) with our participants in a culturally sensitive and appropriate way.

Consider also that the type of information on which  you  place the most value may not match that of people from other cultures. Unrau , Gabor, and Grinnell (2007) [9] point out that mainstream North American cultures place a lot of value on hard data and rigorous processes like clinical trials. (You might notice that we spend a lot of time on this type of information in this textbook.) According to the authors, though, cultures from other parts of the world value relationships and storytelling as evidence and important information. This kind of information is as important and valid as what we are teaching you to collect and analyze in most of this book.

Being the squeaky wheel about evaluating programs can be uncomfortable. But as you go into practice (or grow in your current practice), I strongly believe it’s your ethical obligation to push for evaluation. It honors the dignity and worth of our clients. My hope is that this chapter has given you the tools to talk about it and, ultimately, execute it in practice.

• Ethical considerations in program evaluation are very similar to those in research.
• Culturally sensitive program evaluation requires evaluators to learn as much as they can about cultures different from their own and develop as much cultural awareness as possible.
• Stakeholder input is always important, but it’s essential when planning evaluations for programs serving people from diverse backgrounds.
• Consider the research project you’ve been working on throughout this book. Are there cultural considerations in your planning that you need to think about?
• If you adapted your research project into a program evaluation, what might some ethical considerations be? What ethical dilemmas could you encounter?
• Pruett, R. (2000). Program evaluation 101. Retrieved from https://mainweb-v.musc.edu/vawprevention/research/programeval.shtml ↵
• Mathison, S. (2007). What is the difference between research and evaluation—and why do we care? In N. L. Smith & P. R. Brandon (Eds.), Fundamental issues in evaluation (pp. 183-196). New York: Guilford. ↵
• RAND Corporation. (2020). Step 07: Process evaluation. Retrieved from https://www.rand.org/pubs/tools/TL259/step-07.html. ↵
• Bauer, M., Damschroder, L., Hagedorn, H., Smith, J. & Kilbourne, A. (2015). An introduction to implementation science for the non-specialist. BMC Psychology, 3 (32). ↵
• Aarons, G., Hurlburt, M. & Horwitz, S. (2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services Research, 38 (1), pp. 4-23. ↵
• Stake, R. & Mabry, L. (2007). Ethics in program evaluation. Scandinavian Journal of Social Welfare, 7 (2). ↵
• Unrau, Y., Gabor, P. & Grinnell, R. (2007). Evaluation in social work: The art and science of practice . New York, New York: Oxford University Press. ↵

The systematic process by which we determine if social programs are meeting their goals, how well the program runs, whether the program had the desired effect, and whether the program has merit according to stakeholders (including in terms of the monetary costs and benefits)

individuals or groups who have an interest in the outcome of the study you conduct

the people or organizations who control access to the population you want to study

The people and organizations that have some interest in or will be effected by our program.

A graphic depiction (road map) that presents the shared relationships among the resources, activities, outputs, outcomes, and impact for your program

An analysis of how well your program ended up running, and sometimes how well it's going in real time.

The scientific study of methods to promote the systematic uptake of research findings and other evidence-based practices into routine practice, and, hence, to improve the quality and effectiveness of health services.

The final condition that occurs at the end of an intervention or program.

Tthe long-term condition that occurs at the end of a defined time period after an intervention.

Graduate research methods in social work Copyright © 2020 by Matthew DeCarlo, Cory Cummings, Kate Agnelli is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License , except where otherwise noted.

Share This Book

U.S. Government Accountability Office

Program Evaluation: Key Terms and Concepts

Congress has passed a number of laws to help improve federal management and accountability—including the GPRA Modernization Act of 2010 and the Evidence Act of 2018. These acts aim to strengthen agencies' efforts to build strong evidence to support federal policies.

We developed this glossary to clarify some key concepts that can help federal agencies strengthen the quality of their evidence. It highlights different sources of evidence that can support federal decision-making—such as program evaluations and performance measurement.

This is an update to our previous glossary ( GAO-11-646SP ).

Both the executive branch and congressional committees need evaluative information to help them make decisions about the programs they oversee—information that tells them whether, and why, a program is working well or not. The Government Performance and Results Act of 1993 (GPRA) and GPRA Modernization Act of 2010 (GPRAMA) established a framework for performance management and accountability within the federal government. Building on that foundation, Congress has since passed, among other laws, the Foundations for Evidence-Based Policymaking Act of 2018 (Evidence Act) to strengthen the evidence-building efforts of executive branch agencies. This product updates our previous glossary ( GAO-11-646SP ) to highlight different types of evaluations for answering questions about program performance, as well as relevant issues to ensure study quality. This glossary can help agency officials better understand fundamental concepts related to evaluation and enhance their evidence-building capacity.

For more information, contact Lawrance Evans, Jr. at 202-512-2700 or [email protected] .

Full Report

Gao contacts.

Lawrance L. Evans, Jr. Managing Director [email protected] (202) 512-2700

Office of Public Affairs

Sarah Kaczmarek Acting Managing Director [email protected] (202) 512-4800

Official websites use .gov

A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS

A lock ( ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

Performance and Evaluation Office (PEO) - Program Evaluation

Centers for Disease Control and Prevention. Framework for program evaluation in public health. MMWR 1999;48 (No. RR-11)

This Public Health Reports  article highlights the path CDC has taken to foster the use of evaluation. Access this valuable resource to learn more about using evaluation to inform program improvements.

What is program evaluation?

Evaluation: A systematic method for collecting, analyzing, and using data to examine the effectiveness and efficiency of programs and, as importantly, to contribute to continuous program improvement.

Program: Any set of related activities undertaken to achieve an intended outcome; any organized public health action. At CDC, program is defined broadly to include policies; interventions; environmental, systems, and media initiatives; and other efforts. It also encompasses preparedness efforts as well as research, capacity, and infrastructure efforts.

At CDC, effective program evaluation is a systematic way to improve and account for public health actions.

Why evaluate?

• CDC has a deep and long-standing commitment to the use of data for decision making, as well as the responsibility to describe the outcomes achieved with its public health dollars.
• Strong program evaluation can help us identify our best investments as well as determine how to establish and sustain them as optimal practice.
• The goal is to increase the use of evaluation data for continuous program improvement Agency-wide.

We have to have a healthy obsession with impact. To always be asking ourselves what is the real impact of our work on improving health? Dr. Frieden, January 21, 2014

What's the difference between evaluation, research, and monitoring?

• Evaluation: Purpose is to determine effectiveness of a specific program or model and understand why a program may or may not be working. Goal is to improve programs.
• Research: Purpose is theory testing and to produce generalizable knowledge. Goal is to contribute to knowledge base.
• Monitoring: Purpose is to track implementation progress through periodic data collection. Goal is to provide early indications of progress (or lack thereof).
• Data collection methods and analyses are often similar between research and evaluation.
• Monitoring and evaluation (M&E) measure and assess performance to help improve performance and achieve results.

Research seeks to prove, evaluation seeks to improve. Michael Quinn Patton, Founder and Director of Utilization-Focused Evaluation

E-mail: [email protected]

To receive email updates about this page, enter your email address:

Warning: The NCBI web site requires JavaScript to function. more...

An official website of the United States government

The .gov means it's official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• Browse Titles

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Institute of Medicine (US) and National Research Council (US) Committee on the Review of NIOSH Research Programs. Evaluating Occupational Health and Safety Research Programs: Framework and Next Steps. Washington (DC): National Academies Press (US); 2009.

Evaluating Occupational Health and Safety Research Programs: Framework and Next Steps.

• Hardcopy Version at National Academies Press

2 The Program Evaluation Context 1

P rogram evaluation has been defined as “systematic inquiry that describes and explains the policies’ and program’s operations, effects, justifications, and social implications” (Mark et al., 2000, p. 3) or “… the systematic collection of information about the activities, characteristics, and outcomes of programs to make judgments about the program, improve program effectiveness, and/or inform decisions about future programming” (Patton, 1997, p. 23). The evaluations of National Institute for Occupational Safety and Health (NIOSH) programs carried out under the guidance of the framework committee represent just one way of evaluating research programs. This chapter places the National Academies’ evaluations of NIOSH programs in context by providing a brief overview of the general principles involved in program evaluation and by describing where the process fits in the spectrum of current practices in evaluating research programs. At the conclusion of some of the overview sections throughout the chapter, the committee’s findings specific to the evaluation process used by the framework and evaluation committees are included in bold and italicized text.

• PROGRAM EVALUATION

Although formal program evaluations, especially of educational programs, preceded World War I, the profession as currently practiced in the United States has increased in prominence within the past 50 years. A major impetus to this growth was the need to assess the social programs instituted through the War on Poverty and Great Society policies of the 1960s (Shadish et al., 1991). Legislative requirements for the evaluation of many programs represented a turning point in the growth in the number of evaluations. Evaluation is now an established professional practice, reflected through organizations such as the American Evaluation Association and the European Evaluation Society (AEA, 2009; EES, 2009). Program evaluation is one element of results-oriented management, the approach to public management embodied in the past decade in the Government Performance and Results Act (OMB, 2009a) and the Office of Management and Budget’s (OMB’s) Program Assessment Rating Tool (OMB, 2009b).

Current efforts in program evaluation follow several schools of thought that differ in the evaluation processes used but are all focused on achieving a valid evaluation. The essence of evaluation is determining what is of value in a program. The work revolves around understanding program goals (if available), setting criteria for success, and gathering information to determine whether the criteria are being met as a result of program activities. Program evaluations focus on examining the characteristics of a portfolio of projects rather than assessing one project at a time and often use retrospective information about program outputs and outcomes. Program evaluation differs from a research project in being more tightly connected to practice; it is commissioned by a specific user or organization and designed to inform decision making. It also differs from performance measurement, which is an ongoing process that gathers indicators of what the program is accomplishing but may not assess why the indicators are changing.

Program evaluations can serve several functions. When the program is initially in development or is undergoing changes and is being evaluated with the goal of program improvement, the evaluation is termed a formative evaluation (Scriven, 1991). These evaluations are often initiated and used in-house. When the objective of the evaluation is to assess the program’s outcomes in order to determine whether the program is succeeding or has accomplished its goals, the evaluation is termed a summative evaluation (Scriven, 1967; Gredler, 1996). Users of summative evaluations are often decision makers outside of the program. Program evaluation often also helps communicate the program’s goals and accomplishments to external audiences. Evaluations provide information that contributes to decisions that shape program goals, strategic plans, and actions. In these cases, they serve instrumental functions. Often they also serve enlightenment functions, such as increasing general understanding of program operations, underlying assumptions, or social context (Weiss, 1977).

The practice of evaluating research programs has historically been somewhat separate from that of social program evaluation. Qualitative assessments of research programs in the United States date back to the 1950s (NAS, 1959). The evaluation of research programs took a more quantitative turn in the 1970s as evaluations started to draw on the new availability of large-scale databases to describe scientific activity. Research program evaluation is distinguished from social program evaluation in a number of ways, including the dominant use of peer-review panels and the use of specialized data, including publication and patent-based measures (see discussion later in this chapter).

The evaluations of NIOSH programs discussed in this report were un dertaken in the context of the externally mandated Program Assessment Rating Tool process, a summative evaluation process developed by OMB. However, NIOSH leadership established their primary goal as program improvement, making the evaluations primarily formative.

• LOGIC MODELS

The evaluations of NIOSH programs used logic models—both a general logic model for NIOSH research and specific logic models for each program evaluated. Prior to the work of the evaluation committees, NIOSH contracted with RAND Corporation to provide operational and analytical assistance with compiling the evidence packages for the reviews and developing the logic models; a detailed description of that effort can be found in a recent RAND report (Williams et al., 2009).

Logic models are widely used in program evaluation (W. K. Kellogg Foundation, 2000; World Bank, 2000) to represent visually what evaluators call “program theory.” This phrase refers to the understanding of how the program is supposed to work. How do the program resources become results, and through what channels do those results have their expected impacts? The logic model may be represented as a set of boxes and arrows or as a hierarchy of goals, intermediate outcomes, and final outcomes. The representation provides guidance for the evaluation by pointing to relevant kinds of information to be considered in the assessment and often to indicators in the various areas of the model.

McLaughlin and Jordan (1999) refer to logic models as a way of “telling your program’s performance story.” The common elements of logic models are inputs, activities, outputs, customers, and outcomes (short, medium, and long term), plus external influences (Wholey, 1983; Figure 2-1).

Building a logic model is a process that should involve a team of people with different roles in the program who interact with external stakeholders at many points. After collecting relevant information and clearly identifying the problem the program addresses, the team organizes its information into various elements and composes a diagram that “captures the logical flow and linkages that exist in any performance story” (McLaughlin and Jordan, 1999, p. 68).

Elements of the logic model. Reprinted from McLaughlin and Jordan, 1999, with permission from Elsevier.

Logic models are nearly always predominantly linear and causal because agencies use them to think through how programs will achieve their public goals. In research planning and evaluation, this linearity is ironic. The widespread consensus is that research does not create its effects in a linear fashion. Rather, it is embedded in a complex ecology of relationships that shape and spread knowledge through a variety of channels, not just research knowledge.

Additionally, it is challenging for logic models to capture some outputs such as the development of human capital. Over time, a program may have a significant impact on a field by helping to build a community of practitioners and researchers. For example, NIOSH’s impact on the existence and growth of the occupational safety and health research community is hard to capture in a logic model. In addition, ongoing dialogues with external stakeholders shape research activities and spread research knowledge in ways that are hard to track. Program evaluations that solely rely on the logic model almost inevitably miss information on some of the nonlinear effects of program activities.

The logic models used in the evaluation of NIOSH programs helped pro gram staff and evaluators organize information into steps in the flow of program logic. However, because some of the NIOSH programs spanned several NIOSH divisions and laboratories, the logic model sometimes made it hard for the evaluation committee to grasp the full picture of the research program. Furthermore, the logic models focused a great deal of attention on the most readily observable short- and medium-term outcomes, perhaps missing information on nonlinear and more diffuse contributions of the programs to the development of knowledge and hu man capital in occupational safety and health.

• ROLE OF STAKEHOLDERS

The practice of program evaluation has paid special attention to external stakeholders and the role they play in the evaluation process. Sometimes stakeholders are direct beneficiaries of the program; for example, for a day-care center program, the major stakeholders are the families whose children receive care. Sometimes the stakeholders are organizations with whom the program must work to achieve its goals. In the case of research on occupational safety and health, key stakeholders include workers, employers, and regulatory agencies.

Stakeholder participation in evaluating research programs has come more slowly than in social program evaluation. Early evaluation panels tended to consist entirely of scientists and engineers. But as research policy became more focused on making research relevant to the private sector, evaluation panels also began to include industry and labor representation, often scientists and engineers working in industry and labor organizations. Individuals and families exposed to environmental hazards often organize to increase research and remediation efforts, and stakeholders from these groups also participate in evaluation processes.

Just as social program evaluation pays particular attention to differences in knowledge and expertise between evaluators and stakeholders, in the evaluation of research programs the different contributions of scientific experts and external stakeholders both need to be respected. When the research being evaluated is intended to serve vulnerable populations, current standard practice in the evaluation of research programs, as described in the last paragraph, is not sufficient to give voice to these groups and additional attention needs to be paid to obtaining adequate input.

The National Academies evaluation committees included a variety of members with strong connections to NIOSH’s external stakeholder groups, such as manufacturers of safety equipment, labor organiza tions, and employers. The committees also reached out to a wide range of external stakeholder groups for input, including vulnerable worker populations.

• METHODS OF EVALUATION

Evaluations of research programs necessarily use a variety of approaches. Expert panel review is the “bread-and-butter” approach worldwide, but there is also a long track record of evaluation studies, in which external consultants gather and analyze primary data to inform the expert deliberations.

Within the range of evaluation approaches for research programs, the National Academies’ evaluations of NIOSH programs clearly fall among expert panel evaluations, rather than evaluation studies.

Expert Panel Review

Merit review, peer review, and expert panels are used widely for both ex ante and ex post evaluations of the productivity, quality, and impact of funding organizations, research programs, and scientific activity. Benefits and limitations of this approach have been reviewed extensively (Bozeman, 1993; Guston, 2003; Hackett and Chubin, 2003). Expert panel review is the oldest—and still most common—form of research and development evaluation. In fact, the expert panel is very much a historical development from the National Academies itself, which was established in the 19th century to provide scientific and technical policy advice to the federal government. The underlying evaluative theory of the expert panel is that scientists are uniquely able to evaluate the quality and importance of scientific research (Kostoff, 1997). The preeminence of scientists to evaluate the quality and importance of scientific research was further codified in the research agencies that developed under the philosophy of Vannevar Bush in the 1940s (Bush, 1945).

Expert judgment is particularly capable of evaluating the quality of discrete scientific research activities and the relevance of such discrete activities to particular bodies of knowledge. For example, toxicologists and biochemists—through their scientific training—are uniquely capable of assessing the contributions of particular theories, research methodologies, and evidence to answer specific scientific questions and problems. The major limitation of expert panel review is that traditional training and experience in the natural and physical sciences do not prepare scientists to address questions related to the management, effectiveness, and impact of the types of broad research portfolios that federal agencies typically manage.

Although expert panel reviews work to balance conflicting values, objectives, or viewpoints, they also may lead to tensions in the same areas they are expected to resolve. As noted above, the review process may be broadened to include other stakeholders beyond “experts” or “peers.” Expert panels usually operate with an evaluation protocol developed by an outside group, including evaluation procedures, questions to be answered, and evaluation criteria (e.g., the evaluation of the Sea Grant College Program, Box 2-1 ). The panels usually review a compilation of data on the program, including plans, input counts (budget, staffing), project descriptions, and lists of results. They then talk with individuals connected to the program, both inside and outside the given agency, and use their own experience and judgment in reaching conclusions.

Evaluation of the National Sea Grant College Program. The National Sea Grant College Program, funded by the National Oceanic and Atmospheric Administration, is a nationwide network of 30 university programs aimed at conducting research, education, and (more...)

Closely tied to review processes is the assignment of various types of ratings. For example, the Research Assessment Exercise of the United Kingdom uses 15 panels and 67 subpanels following a common protocol to assess university research programs and assign scores by discipline area (RAE, 2009). Rating scales are be ing used more frequently as evaluations have become more and more oriented to demonstrating performance to outside audiences or to allocating resources. Rating scales capture qualitative judgments on ordinal scales and allow for descriptions of performance at the various levels.

Characteristics that are sought in expert panel reviews include a panel with a balanced set of expertise and credibility among various stakeholder groups and independence and avoidance of conflict of interest among panel members to the extent possible. Selection of panel members can involve trade-offs between recruiting independent reviewers or recruiting reviewers with knowledge and understanding of the program and its field of science. For this reason, expert review panels are seldom completely free of bias and may have conflicts of interest; the preferred practice, of course, is for conflicts to be considered and disclosed. Independence is also reinforced when the panel is commissioned by, and reports to, a user located at least one level above the program in the management hierarchy. The panel adds value by including its perspectives and insights in its report. The panel makes the evidence base for its conclusions explicit in the report and usually makes a limited number of realistic recommendations, phrased broadly enough to allow management to adapt the recommendations to specific circumstances.

The National Academies committees follow a thorough bias and conflict- of-interest process that includes completion of disclosure forms and the bias and conflict-of-interest discussion held at the first meeting.

Other Methods of Evaluating Research Programs

Other types of evaluations generally involve hiring consultants to provide analyses of specific outputs of the program. Because the goal of a research program is new knowledge, publications represent a concrete and observable manifestation of new knowledge and are frequently used as a convenient measure of research program outputs. Publications in peer-reviewed journals provide an indication of quality control, and citations to published articles are used to assess the scientific impact of the work. Patents provide a similar set of measures for technology development. Thus, evaluations of research programs have extensive relevant datasets on which to base their assessments.

Statistical analyses of data on publications (e.g., books, journal articles, review articles, book chapters, notes, letters) range from fairly simple counts and comparisons of publications to highly sophisticated factor analyses and correlations of many types of terms, such as keywords, institutions, and addresses, that lead to the development of networks or maps of the ways in which the research outputs are connected. These bibliometric methods are used extensively to evaluate research activities and compare research output across institutions, disciplines, fields, funding programs, countries, and groups of researchers (Kostoff, 1995; Georghiou and Roessner, 2000; Hicks et al., 2004; Weingart, 2005). Bibliometric methods also can be used to assess the extent of collaboration. Visualization techniques now produce “maps of science” allowing organizations that support research to “see” where the work they have supported fits into research in a specific field or the extent to which it is being used in other research endeavors. An important strength of bibliometric analyses is that they are data-based analyses following a fixed set of rules or algorithms. The analyses are often used as a complement to peer-review techniques, surveys, or impact analyses of research activities. An important weakness, however, is that the measures are incomplete. They do not capture all the dimensions of performance or its context, factors that an evaluation usually needs to consider. In general, a composite set of measures is used to determine the effectiveness of the research activities, institutions, or national programs ( Box 2-2 ).

Review of the National Science Foundation’s Science and Technology Center Programs. Beginning in 1989, the National Science Foundation (NSF) established 25 Science and Technology Centers (STCs) across the United States. The goal was to promote (more...)

Other methods used in evaluating research programs include methodologies drawn from the social sciences, including case studies, interviews, and surveys. One special application of case studies in the evaluation of a research program, for example, is the TRACES approach, named for an early study of Technology in Retrospect and Critical Events in Science (IIT, 1968). This approach starts from a recent accomplishment or success, then tracks the complex set of earlier research results and technologies that made it possible. Programs with economic goals have also used case studies to illustrate the return on investment in advanced technology projects (Ruegg, 2006).

In summary, the evaluation of research programs is an established branch of program evaluation. The National Academies’ evaluation of NIOSH research programs used one of the most common approaches: expert panel review. As is common in evaluations of applied research programs, this process involved stakeholders as members of the evaluation committees and also sought external stakeholder input. The evaluation framework described in Chapter 3 organizes data into a common evaluation tool based on a logic model approach and provides for consideration of external factors. Similar to many research program evaluation efforts, the evaluation committees used this structured rating tool to provide some consistency in ratings across programs. The process did not, however, expand into an evaluation study by gathering new data or extensively analyzing external data sources. The evaluations of NIOSH programs fall well within the range of acceptable practice in evaluating research programs and are compiled in comprehensive reports that went through peer review under the National Academies’ report review process.

• AEA (American Evaluation Association). 2009. American Evaluation Association . http://www ​.eval.org (accessed March 23, 2009).
• Bozeman, B. 1993. Peer review and evaluation of R&D impacts . In Evaluating R&D impacts: Methods and practice . Edited by B. Bozeman, editor; and J. Melkers, editor. . Boston, MA: Kluwer Academic. Pp.79–98.
• Bush, V. 1945. Science: The endless frontier . Washington, DC: U.S. Government Printing Office.
• EES (European Evaluation Society). 2009. European Evaluation Society . http://www ​.europeanevaluation.org/ (accessed March 23, 2009).
• Fitzsimmons, S. J., O. Grad, and B. Lal. 1996. An evaluation of the NSF Science Technology Centers Pro gram . Vol. 1, Summary. Washington, DC: Abt Associates. http://www ​.nsf.gov/od ​/oia/programs/stc/reports/abt.pdf (accessed March 23, 2009).
• Georghiou, L., and D. Roessner. 2000. Evaluating technology programs: Tools and methods . Research Policy 29(4–5):657–678.
• Gredler, M. E. 1996. Program evaluation . Englewood Cliffs, NJ: Merrill.
• Guston, D. 2003. The expanding role of peer review processes in the United States . In Learning from science and technology policy evaluation: Experiences from the United States and Europe . Edited by P. Shapira, editor; and S. Kuhlmann, editor; . Cheltenham, UK, editor. , and Northampton, MA: Edward Elgar. Pp.81–97.
• Hackett, E., and D. Chubin. 2003. Peer review for the 21st century: Applications to education research . Paper presented at the National Research Council Workshop, Washington, DC, February 25, 2003. http://www7 ​.nationalacademies ​.org/core/HacketChubin ​_peer_review_paper.pdf (accessed November 16, 2008).
• Hicks, D., H. Tomizawa, Y. Saitoh, and S. Kobayashi. 2004. Bibliometric techniques in the evaluation of federally funded research in the United States . Research Evaluation 13(2):78–86.
• IIT (Illinois Institute of Technology). 1968. Technology in retrospect and critical events in science . Vol. 1. Chicago, IL: IIT Research Institute.
• Kostoff, R. N. 1995. Federal research impact assessment—axioms, approaches, applications . Scien tometrics 34(2):163–206.
• Kostoff, R. N. 1997. Peer review: The appropriate GPRA metric for research . Science 277:651–652.
• Mark, M., G. Henry, and G. Julnes. 2000. Evaluation: An integrated framework for understanding, guiding, and improving policies and programs . San Francisco: Jossey-Bass.
• McLauglin, J. A., and G. B. Jordan. 1999. Logic models: A tool for telling your program’s performance story . Evaluation and Program Planning 22:65–72.
• NAPA (National Academy of Public Administration). 1995. National Science Foundation’s Science and Technology Centers: Building an interdisciplinary research program . Washington, DC: NAPA.
• NAS (National Academy of Sciences). 1959. Panel reports of the NAS-NRC panels advisory to the National Bureau of Standards . Washington, DC: National Academy of Sciences.
• NRC (National Research Council). 1994. A review of the NOAA National Sea Grant College Program . Washington, DC: National Academy Press.
• NRC. 1996. An assessment of the National Science Foundation’s Science and Technology Centers Program . Washington, DC: National Academy Press.
• NRC. 2006. Evaluation of the Sea Grant Program review process . Washington, DC: The National Academies Press.
• OMB (Office of Management and Budget). 2009. a. Government Performance and Results Act . http://www ​.whitehouse ​.gov/omb/mgmt-gpra/gplaw2m.html (accessed March 20, 2009).
• OMB. 2009. b. Assessing program performance . http://www.whitehouse. gov/omb/part/ (accessed March 20, 2009).
• Patton, M. Q. 1997. Utilization-focused evaluation: The new century text . Thousand Oaks, CA: Sage.
• RAE (Research Assessment Exercise). 2009. Research Assessment Exercise . http://www ​.rae.ac.uk/ (accessed March 23, 2009).
• Ruegg, R. 2006. Bridging from project case study to portfolio analysis in a public R&D program—A framework for evaluation and introduction . NIST GCR 06-891. http://www ​.atp.nist.gov ​/eao/gcr06-891/gcr06-891report.pdf (accessed March 23, 2009).
• Scriven, M. 1967. The methodology of evaluation . In Curriculum evaluation: AERA monograph series on evaluation . Edited by R. E. Stake, editor. . Chicago: Rand McNally. Pp.39–85.
• Scriven, M. 1991. Beyond formative and summative evaluation . In Evaluation and education: At quarter century . Edited by M. W. McLaughlin and, editor; D. C. Phillips, editor. . Chicago: University of Chicago Press. Pp.19–64.
• Shadish, W. R., T. D. Cook, and L. C. Leviton. 1991. . Foundations of program evaluation: Theories of practice . Newbury Park, CA: Sage.
• Weingart, P. 2005. Impact of bibliometrics upon the science system: Inadvertent consequences? Scientometrics 62(1):117–131.
• Weiss, C. H. 1977. Research for policy’s sake: The enlightenment function of social research . Policy Analysis : 3:531–545.
• Wholey, J. S. 1983. Evaluation and effective public management . Boston: Little, Brown.
• Williams, V. L., E. Eiseman, E. Landree, and D. M. Adamson. 2009. Demonstrating and communicating research impact: Preparing NIOSH programs for external review . Santa Monica, CA: RAND.
• W. K. Kellogg Foundation. 2000. Logic model development guide . http://www ​.wkkf.org/Pubs ​/Tools/valuation/Pub3669.pdf (accessed March 23, 2009).
• World Bank. 2000. Logframe handbook: A logical framework approach to project cycle management . http://www ​.wau.boku.ac ​.at/fileadmin/_/H81 ​/H811/Skripten/811332 ​/811332_G3_log-framehand book.pdf (accessed March 23, 2009).

This chapter draws on background papers commissioned by the committee from Sonia Gatchair, Georgia Institute of Technology, and Monica Gaughan, University of Georgia.

• Cite this Page Institute of Medicine (US) and National Research Council (US) Committee on the Review of NIOSH Research Programs. Evaluating Occupational Health and Safety Research Programs: Framework and Next Steps. Washington (DC): National Academies Press (US); 2009. 2, The Program Evaluation Context.
• PDF version of this title (3.6M)

In this Page

Recent activity.

• The Program Evaluation Context - Evaluating Occupational Health and Safety Resea... The Program Evaluation Context - Evaluating Occupational Health and Safety Research Programs

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

Connect with NLM

National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894

Web Policies FOIA HHS Vulnerability Disclosure

Help Accessibility Careers