Program Evaluation

Program Evaluation

The purpose of conducting these program evaluations is to improve our programs and decide where to put resources. Are programs implemented as intended? Are current programs meeting member needs? Where can we best put our resources for best ROI in serving our mission? How do I identify the stakeholders for this particular program? The Board Officers of the Nevada Occupational Association will be the recipients of the evaluation results.

Introduction

Overview

Logic Models

I want to learn more!  Where can I go?
What should I make my logic model in?
  • Powerpoint, Google Drive, Word 
  • LucidChart (there are free and paid plans)
  • Pen + Paper, white boards, post-it notes!
What about indicators?  

Possible Evaluation Methods

There is no one perfect method to suit all program evaluations or needs assessments. We will complete these in approximately 60 days so that means the method should fit that timeframe. If you have recommendations for more longitudinal evaluation, please put that in your recommendations.

Survey Designs

Please use Google Forms.  No Survey Monkey, please.  Also, Typeform may be an option  – check it out and let us know what you think!

Remember, response rates can be low.  Consider how to make surveys quick and easy to complete.

Focus Groups

You should always have a partner with you during a focus group so one can facilitate discussion and the other can take notes.

Document Review and Counts

Other Review Methods

You can use a combination of the above methods or any other method you find appropriate for your evaluation.

Frequently Asked Questions

 

Q: I designed a survey/interview but am not hearing back from people. What should I do?

A: Many people benefit from a friendly reminder. Give them a deadline and if they don’t respond by then, consider another way to collect what information would be useful to you.

Q: Is this research?

A: No, this is not research. The purpose is to improve our programs. We still want to maintain confidentiality and ensure participation is voluntary, but there is not informed consent or a research protocol.

Q: I am doing a document review but find there is missing data, not all documents have the information I am looking for. What should I do?

A: This is normal in document reviews. Decide how to handle missing data. Often you can just drop that particular indicator. Sometimes it makes sense to estimate or substitute a mean for that missing data, but for these purposes just report on what data you do have and report how much was missing and how you think that affected the understanding of outcomes.

Q: I am doing interviews/focus groups. Can I record them?

A: This depends. You have to get consent from the interviewee/participants. If this is via Zoom, they can consent on Zoom. If this is in person, you would have to create a written consent. Consider if recording the interview will affect their responses. It may, depending on your questions. Also, consider how to dispose of the recording once completed.

Q: What will be useful to the board at the end of the evaluation?

A: A presentation to the NOTA board with the following information, complete with visuals:

  • A logic model for your program/s, including goals, assumptions. If one is not already made, please make one.
  • Report showing what you found during the evaluation/assessment/analysis, including recommendations for moving forward
  • For existing programs:
    • Was it implemented as intended?
    • Did it produce the desired outputs and outcomes? Why or why not?
    • How and why is it working or not working?
    • Please note these are not the only questions – there will be discussion with the board.
  • For future programs:
    • What do members want?
    • When/how should we implement this?
    • What resources/inputs will be needed for this?
    • What is realistic for us? What opportunity cost is involved?