Detailed Project Description

The PACE Project builds on the solid experience and expertise of the project team and builds on our prior work. The following section presents the specific Project activities.

Participants

Participants in this Project will include two staggered cohorts of program staff (two staff per program) representing a total of 16 programs. Approximately eight programs will be assigned to cohort 1 and eight programs will be assigned to cohort 2. During recruitment, the Project will target promising school and non-school based youth programs (e.g., scouting, athletic organizations, religious organizations, etc.) that emphasize the development and strengthening of character and virtues such as honesty, generosity, forgiveness, diligence, thrift, joy, curiosity and humility. (Programs might or might not self-identify as youth character development programs.)
Each program will be asked to submit an application to participate in the Project. The application will include a description of the program as well as any current evaluation work and a statement of interest. This information will be used to assess readiness to participate in the Project activities laid out below as well as the potential benefit of each applicant program’s participation. For the purpose of this Project, “readiness” will be defined as having a clear set of youth character development program activities and some defined youth character development outcomes (i.e., it must be a youth character development program), as well as motivation to participate in a program and evaluation planning partnership (i.e., participants must have interest in working with an evaluator to develop a program model and potentially plan an evaluation that they themselves would implement). In particular, we are looking for: program implementers who have an intrinsic motivation to improve the way their program plans, implements, and uses evaluation, and who currently faces a lack of support and resources in the area of evaluation). Programs invited to participate will receive a $2,500 stipend plus travel expenses during their 12 (cohort 2) – 15 (cohort 1) month commitment to the Project. In addition, any program team that chooses to develop a complete evaluation plan and implement that plan in partnership with their partner evaluator will receive an additional $2,000 which can be used to offset the cost of this additional evaluation work. In the case that we have more high-quality applicants that we can accept, we will seek to maximize the diversity of programs (in terms of area of focus, format, and approach to evaluation) among participants.

We will also recruit one cohort of approximately eight evaluators to participate in the Project. This group will comprise individuals with training and experience in the field of program evaluation. Experience specifically in evaluation capacity building (ECB), Evolutionary Evaluation (EE), promoting Evaluative Thinking (ET), the Systems Evaluation Protocol (SEP), and/or Person-Centered Analysis (PCA) is not necessarily required. During recruitment, the Project will be seeking individuals who are currently working and/or studying in the field of evaluation including, but not limited to: evaluation consultants, evaluation capacity builders, and graduate students pursuing a career in evaluation. Similar to participating programs, evaluators will also be asked to submit applications to participate in the Project, including a CV and a statement of interest. Selection criteria will include experience in evaluation, including knowledge of social science research methods, interest in learning about ECB, EE, ET, the SEP, PCA and youth character development programs, motivation to work in partnership with programs (rather than in a more traditional internal or external evaluator role), and interest in remaining part of a network of youth character development program evaluators after the Project. We will seek evaluators who have working knowledge of evaluation methods and concepts, and who are engaged in evaluation practice in some professional capacity. Selection based on experience and expertise will be designed to be inclusive of a range of evaluation knowledge, skills, and experiences, while ensuring that selected evaluators will be able to support a program in the partnership model. Selected evaluators will be offered a $6,000 stipend plus travel expenses for their 15 month commitment to the Project and to their partnerships with up to 2 participating programs. Evaluators who continue their partnership with a program to develop a formal evaluation plan will receive an additional $2,500 stipend for each evaluation plan completed with a program partner.

Project Staff

All Project participant activities will be planned, implemented and/or supervised by a team of ECB professionals with expertise in Evolutionary Evaluation (EE), Evaluative Thinking (ET), the Systems Evaluation protocol (SEP) and Person Centered Analysis (PCA). The facilitation team will be led by the Project Leader (Dr. Jennifer Brown Urban) and co-Project Leader (Dr. Miriam Linver), and will also include two Lead Facilitators (Dr. Monica Hargraves and Jane Buckley) as well as up to 6 additional master ECB facilitators who have prior experience implementing ECB events and activities. The Lead Facilitators developed all in-person workshops and conference activities specifically for the Project, and will be involved in all of the planned events over the course of the Project. The additional members of the Facilitation Team will assist with training activities. In addition, each Lead Facilitator will directly mentor half of the participating programs and half of the participating evaluators, consulting with each at least quarterly during their participation in the Project. To address the need for training in youth character development, Dr. Richard Lerner  will develop and provide training to both program and evaluation participants on youth character development terms, concepts and approaches. To address the need for training in PCA, Dr. Sara Johnson will develop and provide training to evaluation participants.

In addition to the Facilitation Team, the Project will be supported by the work of an expert team of researchers at the REDSS lab at Montclair State University (“Project Evaluation Team” or PET) to evaluate the activities and short-term outcomes laid out in the evaluation plan section of this proposal. The PET will plan and oversee all data collection and analyses related to the internal evaluation of this Project, as well as participate in the planning and supervision of the Project itself. The Project will also be advised by a board of experts including leaders in the fields of ECB and youth character program development and evaluation. The Project Advisory Board will meet annually (3 times) during the course of the Project, and will offer feedback on progress toward Project goals.

Capacity Building for Program Staff

Programs selected to participate in the PACE Project will participate in four types of activities: in-person events, webinars, partnership work with a participating evaluator, and consultation with a Lead Facilitator.

In-person Events

The in-person events for program staff include two workshops (WS1 and WS2) for each program cohort and one Project-wide culminating conference. The workshops will be facilitated by the Project Leader, two Lead Facilitators, and the Facilitation Team. The two program cohorts will be staggered so that program cohort 2 will participate in the workshops 2-3 months after cohort 1 (see table 1 below). WS2 for cohort 2 will be held virtually, to reduce travel costs, and will rely in part on recordings of WS2 for cohort 1. Project staff will seek to assign programs to cohort 2 who would benefit most from a “virtual workshop” – those that are highly motivated and/or most concerned about the time away from work required to participate in an in-person workshop.

The first workshop (WS1) is a three-and-a-half day event devoted to accomplishing three goals:

1. Strengthen the identity of participating programs as youth character development programs, including program staff’s ability to communicate effectively about their contribution to developing specific character virtues. (For example: as part of the pathway model building process, program teams will be asked to identify, review and sharpen the articulation of outcomes related to specific character virtues.)
2. Motivate program staff participants to think evaluatively and incorporate specific Evaluative Thinking habits into their work
3. Increase participating programs’ capacities to plan and implement high quality evaluation work that is based on the principles of EE and the SEP

To accomplish these goals, the Facilitation Team will draw from a large set of well-developed techniques and materials that center around an active-learning approach to ECB. This will include activities such as structured group work, peer-review, large group discussion, role-play/case study, and brainstorming. An agenda for WS 1 is outlined in Table 2 below:

Day 1

Day 2

Day 3

Day 4

As a result of participating in this workshop, program teams (2 program staff per program) will: develop program-specific pathway models (PMs), Program and Evaluation Profiles (PEPs), Evaluative Thinking (ET) site plans, and establish a partnership with a participating evaluator. Brief definitions of the items program teams will produce during workshop 1 are listed in Table 3 below:

Pathway Model (PM)

A pathway model is a type of logic model. It is a graphical representation of the relationships between the activities and outcomes that make up a program. Pathway models communicate the “story” or “theory of change” of a program. They are the essential foundation for determining the scope and questions that guide the evaluation of the program being modeled.

Program and Evaluation Profile (PEP)

The PEP is a document that provides a complete, yet concise “snapshot” of the youth character development program as well as that program’s evaluation focus and goals. The PEP includes: a program description, a copy of the pathway model, a description of relevant stakeholders, context and assumptions, an evaluation purpose statement, and evaluation questions (the broad questions that will guide the evaluation of the program).

Evaluative Thinking (ET) Site Plan

An ET site plan is a document that records specific activities and habits that a program team intends to adopt in order to promote Evaluative Thinking in their home office/organization. ECB facilitators will retain a copy of this site plan in order to check in with programs during the Project.

At the end of WS1 program teams (program staff and evaluator partners) will be asked to consider whether they would like to conclude their partnership work by refining and finalizing the 3 deliverables described above, or to go further and work together to develop and possibly implement formal evaluation plans. Program teams that do not continue their partnership through evaluation plan development and possibly implementation will still participate in WS2 (focused on utilizing results) and the culminating conference. In cases where the program team is interested in continuing on to evaluation planning, they will be asked to submit a simple application and Memorandum of Understanding that describes their proposed scope of work prior to their first phone consultation meeting with a Lead Facilitator (see Consultation with Project staff below). The Lead Facilitator will assess the team’s readiness for formal evaluation planning and offer feedback in the first consultation. At the end of that meeting, teams will either be selected to receive the additional evaluation work stipend, or will be advised to revise their proposed scope of work and re-submit. All program teams will have the opportunity to pursue this work. The selection process is to ensure readiness to develop and implement high quality, life-cycle appropriate evaluation plans that have a high likelihood of being completed.  Lead Facilitators will continue to check in with program teams at least quarterly, and more often as needed for the duration of the Project.

The second workshop (WS2) is a one-and-a-half day event focusing on analysis and utilization of evaluation results. For Cohort 1, WS2 will be in-person; for Cohort 2, WS2 will be held virtually, via web conference. The goals of this workshop are to build participants’ capacity to:

1. Read and make meaning from evaluation results
2. Facilitate the analysis process
3. Make informed decisions in planning for program evolution and reporting

This workshop will also reinforce previous learning in the areas of EE, ET and the SEP. For those programs that have already completed some data collection (as part of current or previous evaluation work), this workshop will offer the opportunity to practice Evaluative Thinking and analysis using their own data. For programs that have not collected any evaluation data yet, case studies will be provided so that these participants can also learn experientially. (Programs do not have to have a complete, formal evaluation plan to participate in this workshop.)

The Project-wide culminating conference will be held at the end of year 2 of the project. All participating program staff and evaluators will be asked to attend. The conference will showcase presentations by the participating youth character development programs and will provide networking opportunities that will strengthen connections among youth character development practitioners, interested and trained evaluators, and prospective funders. As such, it will promote more effective youth character development programming, both through practical connections and by embodying an evaluative culture. The PACE culminating conference is described in more detail in a dedicated section below.

Training Webinars

In addition to the workshops and culminating conference, training will also be offered via webinar. Dr. Lerner will provide training to both program and evaluator participants on youth character development terms, concepts and approaches offered as a webinar before WS1. The webinar will focus on the myriad ways character is defined, available tools for measuring character virtues, how these tools should be used, and how to interpret their use. Dr. Lerner will also discuss how engagement with participants depends on understanding the programmatic theory of change and where and how measurement tools fit in. The goal of the webinar is to help strengthen the identity of participating programs as youth character development programs and gain a shared understanding of youth character development concepts and definitions. In addition, Dr. Sara Johnson will offer a series of 2 webinars on PCA for evaluators and interested practitioners which will take place before WS2. The goal of the webinars is to provide an in-depth introduction to PCA including examples for application in a youth character development – program setting.

As mentioned above, WS2 for Cohort 2 will also be offered virtually (as a webinar). The Facilitation Team may also offer additional webinars over the course of the Project based on emergent needs or interests.

Program-Evaluator Partnership Work – Program Perspective

Programs will be matched with evaluators at the end of WS1. Because there will be twice as many programs as evaluators, each evaluator will be matched with two programs, one from each cohort. Prior to WS1, facilitators will create a directory of both program and evaluator participants that will include information such as: location, content area/specific character virtues, specialties, and methodological interests. Along with this list, all participants will be provided with a brief description of the partnership approach and the intended roles within the partnerships (program staff as experts on the program and implementers of the evaluation, evaluators as experts in methodology and advisors to implementation). All participants will be asked to review this document prior to WS1 with “match making” in mind. There will be time set aside during WS1 for participants to get to know each other and self-select into partnerships. Facilitators will offer advice and assistance to participants as needed to ensure that they are satisfied with their selections. For the remainder of their participation in the Project, the program-evaluator partnerships will work as a unit to develop a program pathway model and PEP, and ET site plan. (If interested and selected, the program-evaluator partnership will be extended to complete a more formal evaluation plan.) Though program pathway models, PEPs, and ET site plans will be developed as part of WS1, follow-up review by stakeholders and additional revisions to these items after WS1 will be encouraged. Therefore, partnership work with evaluators is likely to include phone and/or online meetings outside of scheduled in-person events.

In cases where program staff are interested in developing and/or implementing a more formal evaluation plan, their evaluator partner will serve as their primary source of support for this work. Since resources for evaluation and hiring an evaluator are typically difficult for programs to obtain, the PACE project design will essentially be providing a subsidy to this cost of evaluation by offering a stipend to evaluator partners of $2,500 for each evaluation plan completed with a program partner. Evaluators will also receive additional training to support them in this work. In addition, programs that are selected to continue on to implementation will also receive a $2,000 stipend that can be used to offset the costs of evaluation implementation.

Finally, in preparation for the PACE culminating conference, all program staff will work directly with their evaluator partner to prepare a presentation of their PEP and evaluation work to date.

Consultation with Project Staff

Participating program staff will be encouraged to use their partnering evaluator as their primary source of expertise on evaluation concepts and methods. However, each program and program-evaluator partnership will be guided by one of the two Lead Facilitators. The assigned Lead Facilitator will check in with program staff at least quarterly, and more often as needed, during their participation in the Project (including any implementation work) to answer questions, provide feedback, support the participant/evaluator partnership and collect information needed for the evaluation of the Project. These check-in meetings will take place via phone/web conference and will be scheduled prior to the first workshop for all participating programs. Partner evaluators will also attend these meetings both as program team members and as ECB facilitation apprentices (see Capacity Building for Evaluators section below). Programs will receive additional support from Lead Facilitators on an as-needed basis.

Capacity Building for Evaluators

The experience for evaluators participating in the Project is best described as an apprenticeship. Though participating evaluators will receive initial training in youth character development, EE, ET, the SEP and PCA, most of their learning will be “hands-on” in that they will be asked to help facilitate ECB activities as well as form real partnerships with participating programs. Like participating program staff, evaluators will participate in four types of activities as part of the Project: in-person events (including a facilitation apprenticeship), webinars, partnership work with a participating program, and mentorship with a Lead Facilitator.

In-person Events

Participating evaluators will attend both of the workshops (WS1 and WS2) for both of the cohorts (3 in-person workshops in total) as well as the PACE culminating conference. In addition, there will be a pre-workshop training just for participating evaluators to be held the day before WS1 for cohort one begins (see Table 4).

The day-long pre-workshop training, along with assigned readings to be completed prior to the training, will provide participating evaluators with basic background knowledge of youth character development,  EE, ET, and the SEP. Dr. Richard Lerner will attend the pre-workshop training to present youth character development concepts to evaluators in person. This session will cover the theoretical bases for youth character development, innovations in methodological approaches to evaluating youth character development programs, character as a developmental phenomenon, and features of high quality evaluations of youth character development programs. This introduction to youth character development ECB will also prepare evaluators to be partnered with programs from both cohorts as part of the “program-evaluator partnership work” described below.Evaluators’ role and participation in the ECB workshops will be distinct from that of program staff. In fact, evaluators will have two roles while attending the ECB workshops: apprentice facilitator and program “team” participant (see “program-evaluator partnership work”).

Webinars

Evaluators will participate in the same webinars as participating program staff: one before WS1, presented by Dr. Richard Lerner, which will provide an overview of youth character development terms, concepts and approaches; as well as any additional webinars arranged as needed over the course of the project. In addition, Dr. Sara Johnson will offer a series of 2 webinars on PCA for evaluators which will take place before WS2. The goal of these webinars is to provide an in-depth introduction to PCA including examples for application in a youth character development – program setting.

Facilitation Apprenticeship 

During Cohort 1’s workshops, evaluators will be asked to carefully observe and reflect on the facilitation work of Project staff. This observation, along with mentorship and planning meetings with Lead Facilitators, will prepare evaluators to co-facilitate Cohort 2’s workshops. The purpose of this apprenticeship is to prepare participating evaluators to facilitate ECB activities and workshops for youth character development programs independently in the future and is a critical part of earning their Montclair State University credential as a PACE evaluator. Activities that participating evaluators will be asked to co-facilitate will focus around promoting ET, building pathway models, preparing PEPs, and making meaning through analysis (WS2).

Program-Evaluator Partnership Work

As described in the Capacity Building for Program Staff section, program staff from each program will be partnered with an evaluator for the duration of their participation in the Project. Each evaluator will be partnered with 2 programs, one from each cohort. These partnerships will be formed during WS1, based on mutual interest, complementary needs, and geographical location. Information about each program and evaluator will be made available to all participants prior to WS1 in order to prepare for forming these partnerships. During workshops, evaluators will sit with their program team partners and participate in group work activities (including developing pathway models and PEPs) along with program staff. As a relative outsider to the program, evaluators will offer a unique perspective on the program, but will act as members of the program evaluation team rather than external evaluators. In between workshops, evaluators will continue to work with their program staff partners to develop and revise PMs, PEPs, and complete any additional evaluation work as needed and appropriate. Participating evaluators will also attend scheduled quarterly phone/online meetings with their partner programs and the Lead Facilitator supervising that program (as described in the “Program-Evaluator Partnership Work – Program Perspective” section). These meetings will provide an opportunity for Project staff to monitor and support program-evaluator partnerships and contribute to the evaluator’s development as an apprentice ECB facilitator. Evaluators selected to extend their partnership work to include formal evaluation planning will be expected to meet with their program teams at least monthly, and more often as needed to complete a formal evaluation plan. In preparation for this additional work, selected evaluators will also attend one additional meeting with a lead facilitator to prepare for the planning process.

The PACE Culminating Conference

At the end of year two of the PACE Project (Fall, 2017), all facilitators and participants (program staff and evaluators) will be invited to attend a 2-day conference hosted by Project staff in a location that maximizes accessibility and minimizes travel costs based on the home location of selected programs and evaluators. This conference will be the culminating event for the Project and will serve as an opportunity to share the work of participants as well as the Project as a whole with internal stakeholders and the broader youth character development, ECB, program and evaluation communities. The conference will have 4 main components: the PEP poster session, evaluator presentations and awarding of credentials, program evaluation presentations, and Character Virtue team meetings. Each of these components is described separately below.

Evaluator Presentations

Participating evaluators will be asked to give a presentation on their approach to evaluation and evaluation capacity building as they relate to youth character development programs. This presentation should be a consolidated representation of their learning over the course of the Project and is also an opportunity for each evaluator to market themselves and their services to the youth character development community. At the end of their presentations, each participating evaluator who has completed the Project will receive a PACE Project credential from Montclair State University. In addition, all PACE Project evaluator “graduates” will have their name and contact information added to a publically available list on the John Templeton Foundation and REDSS Lab (Montclair State University) websites.

PEP Poster Session

At the PEP poster session each participating program will present a poster version of their PEP, which will include their program description, pathway model, evaluation purpose statement and questions, along with other relevant information. Fellow conference attendees will be able to view the PEP posters and get a quick yet accurate idea of what the program is about and its specific evaluation goals. To encourage networking, this poster session will be offered as a cocktail hour event and food will be provided. In addition, each program team will receive a certificate of participation in the PACE Project.

Program Evaluation Presentations

For those programs that have completed an evaluation or some evaluation work since beginning their participation in the Project, program staff will be offered the opportunity to present that work in the form of a short (10-15 minute) presentation. Programs interested in presenting will submit their work to Project staff for review before being formally invited to present. These presentations will offer another opportunity for program staff to connect with other evaluators as well as serve as a potential model for peer programs.

Character Virtue Team Meetings

At the end of the PACE Project, there will be networks of exemplar youth character development programs, each with the capacity to clearly communicate their specific contributions to youth character development and the evaluation of these contributions. In order to create connections across cohorts and sustain this larger network, we will ask programs to organize themselves around the primary character virtue(s) their program promotes and meet as “character virtue teams” to share ideas and offer support. For example, if there are six participating programs with a strong focus on humility, these programs will meet as a larger group to discuss program design and evaluation topics specific to humility. Evaluators who are interested or have expertise in a particular virtue will also be invited to these team meetings to learn more about participating programs and share expertise, resources and contact information with the group.