A Revised Version of Assignment # 1

Assignment #1 A completed evaluation Case: Client Satisfaction Survey Analysis, Quebec Regional Office, Human Resources Development (HRDC) in Quebec, Canada, 2001.

The study is an analysis of requirements and proposal for a client satisfaction measurement program. Circum Network Inc. is listed as the author of this study. It was prepared for Evaluation Services Information and Strategic Planning Directorate Quebec Regional Office Human Resources Development.

Model or process used in the evaluation: An improvement-focused model (Posovac, 2003, p.29). While providing the essential methodological foundations, “the self-directed” training document takes a pragmatic and “integrated” approach to conducting client satisfaction surveys. It includes devices such as decision trees and checklists. The project was carried out in three phases. First, the team developed the standardized questionnaires. Then the researchers developed an operational framework for the client satisfaction measurement program. Finally, they developed the analytical support tools and mechanisms.According to the report, there is a consensus within the HRDC-Quebec Region that systematic, rigorous measurement of client satisfaction with the products and services offered by the Region is essential to building the fifth pillar of the region’s vision: delivering services of the highest quality. There is also a consensus that the primary responsibility for improving the region’s services lies with the HRCCs and other operational centres, because it is they that control the daily delivery of services.

Strengths: (1) The goals were to plan a client satisfaction measurement program and an analysis of requirements and proposal for a client satisfaction measurement program; the team produced a self-directed training document on the implementation of client surveys. (2) The report presents clearly the development of standardized questionnaires, operational and implementation framework and pre-testing. (3) The report is a Guide for human resources development (employees who do not necessarily have the knowledge required to conduct formal, systematic surveys). The tools offered in the guide are to be used for measuring, sampling, collecting data, analyzing data, interpreting results, and implementing recommendations. The results included standardized questionnaires for various types of clients and various services’ conditions. (4) Researchers/evaluators helped programs’ staff to learn how to discover discrepancies between program objectives and the needs of the target population, between program implementation and program plans, between expectations of the target population and the services actually delivered, or between outcomes achieved and outcomes projected (Posovac, 2003, p.29).
In this sense, I think in this case, it was used an approach similar to Stake's countenance model explained by Jay Wilson (2009) because there was a need for formalized evaluation. It was not just anecdotal but descriptive data was necessary. It included description and judgment, intents, and observations, which were compared to standards then a judgment was made. In short, there was a "mix" or mixture of parts pieces (quantitative/qualitative elements) and, as a result, it was an “artistic” evaluation (creative thinking in the minds of evaluators).
Regarding possible weaknesses, I see a couple of things that could be considered in future program evaluations: (1) the program does not compare or discusses all sides of program evaluation both positive and negative. Deliberation of pros and cons are not evident in the survey evaluation of the program. It would be useful if this discussion or deliberative process can take place. (2) It does not describe participant’s “reactions” to and “learning” from the innovative program evaluation, as well as “behavior” changes in real job performance, and other potpourri “results.” A round table or discussion of these issues could help to enlightening program evaluation.

Sources:
Circum Network Inc. (2001) An integrated approach to conducting client satisfaction surveys analysis of requirements and proposal for a client satisfaction measurement program. Prepared for Evaluation Services’ Information and Strategic Planning Directorate Quebec Regional Office Human Resources Development Canada.
Retrieved September 7, 2009 from:
http://www.circum.com/textes/program_hrdc_quebec_2001.pdf

David Crawford (2009) Evaluation exploration. Retrieved September 4, 2009, from http://www.ag.ohio-state.edu/~brick/evalexpl.htm

Miller, R., & Butler, J. (2008) Using an adversary hearing to evaluate the effectiveness of a military program. The Qualitative Report, 13(1), 12-25. Retrieved September 5, 2009 from http://www.nova.edu/ssss/QR/QR13-1/miller.pdf

Posovac, E., & Carey, R. (2003). Program Evaluation: Methods and Case Studies. (6th edition). New Jersey: Prentice Hall.