National 4-H Science Leadership Academy

Year Two Regional 4-H Science Academies Evaluation Report
June 15, 2012 Mary E. Arnold, Ph.D. Project Evaluator Oregon State University
With assistance from

Courtney Archibeque, MpH Brooke Nott, M.S. Graduate Research Assistants Oregon State University

Table of Contents
Evaluator’s Statement Acknowledgements Executive Summary ………………………………………………………………….. ii ………………………………………………………………….. iii ………………………………………………………………….. iv

Year Two Regional 4-H Science Academy Overview …………………………………………………….. 1 Knowledge and Skills for 4-H Science: Impact of the Regional Academies …………………….. 8 Curriculum Focus ………………………………………………………………..… 8 Evaluation Focus ………………………………………………………………….. 10 Professional Development Focus …………..……………………………………………………... 11 Fund Development Focus ………………………………………………………………….. 12 Summary ………………………………………………………………….. 14 Confidence and Intention to Teach Others Curriculum Evaluation Professional Development Fund Development Readiness to Facilitate 4-H Science Summary ………………………………….……….. 15 ………………………………………………………………..… 15 ………………………………………………………………….. 18 …………..……………………………………………………... 21 ………………………………………………………………….. 23 ………………………………………………………………….. 26 ………………………………………………………………….. 27 ………………….………….... 28 …………….……… 28 …………..……….. 29

Science Liaisons Estimated Reach Beyond Regional Academies Estimates of Knowledge and Skill Dissemination Additional Resources Garnered for 4-H Science

Narrative Questions Content Analysis …………………………………………... 33 Plans for Next Three Months ………………………………………………………………..… 33 Additional Support Needs ………………………………………………………………….. 35 Post-Academy Training Opportunity Ideas………………………………………………………….. 36 Appendix 1: Feedback from New England on Effectiveness of Virtual Academy ..……... 39

i

Evaluator’s Statement
This document serves as the final evaluation report for the National 4-H Science Academy: Year Two Regional Academies program sponsored by National 4-H Council with funding from the Noyce Foundation. The academies were held in five locations across four regions of the country between January and April, 2012. All academy participants were invited to participate in the program evaluation. All data for the evaluation were entered by participants directly into an on-line data collection system. Access to the system was provided by the evaluator to the participants for data entry, but only the evaluator and her research assistants had access to the actual dataset. The integrity and accuracy of the raw data rests with the individual participants. The integrity and accuracy of the analysis and interpretation rests solely with me as the project evaluator. To this end, I certify that the analysis and results presented in this document are complete and accurate insofar as the data entered by the participants were as well. Any questions or concerns about this report should be addressed to me.

Mary E. Arnold, Ph.D. Project Evaluator, Oregon State University June 15, 2012 mary.arnold@oregonstate.edu

ii

Acknowledgements
This program evaluation could not have taken place without the dedicated help and support of many individuals. First and foremost, I would like to thank National 4-H Council and the Noyce Foundation for the opportunity to conduct the Year Two Regional Academy evaluation. It was exciting to witness the movement from the national to the regional level, and to document the important work that took place as a result of the academies. I would like to express my thanks and gratitude to Janet Golden, Eddie Locklear, Jo Turner, and Maila Oliveria at National 4-H Council for their support and help with various aspects of the evaluation. Thank you, also, to the lead planners at each of the five academies, for keeping me apprised of the plans for your academies, and more than anything for your timely assistance with getting participant contact information to me. A very special thank you to my co-authors and graduate assistants: Ms. Courtney Archibeque and Ms. Brooke Dolenc Nott. Your focused assistance with the data analysis and report preparation could not be replaced. Thank you both especially for your cheerful willingness to concentrate your work time on the report so we could meet the expected deadline. I would like to thank each and every academy participant who contributed data. The sincerity with which you approached the evaluation was evident in the data and information you provided. Without your help, there would be nothing to report. So thank you for your willingness to help make this possible. Finally, thank you to the Noyce Foundation for the generous support of the National 4-H Science Leadership Academy. The funding provided by the foundation made this important program possible. As a result, 4-H programs across the country are more prepared to develop and sustain programs for youth in science, technology, engineering, and math.

iii

Executive Summary
Year Two Regional 4-H Science Academy Overview  Five regional 4-H Science Academies were held during the second year of the National 4-H Science Leadership Academy project: o Northeast (Mid-Atlantic), State College, PA, January 24-26 (73 participants) o North Central-St. Louis, MO, February 7-10 (228 participants) o West-Davis, CA, March 26-28 (100 participants) o Southern- Huntsville, AL, April 9-12 (78 participants) o Northeast (New England), Virtual Academy, April 10-11 (36 participants)  515 invitations to participate in the academy evaluation were sent, with 427 returned, of an overall return rate of 82.9%:  Demographic data from the regional academies indicate that the primary audience for the academy (county 4-H agents) was reached.  Academy attendance overall appears to represent the size and involvement of 4-H Science programs for each region.  Participants report a fairly even exposure to the academy topics, with slightly more attending sessions on curriculum and professional development than the other areas. This, however, matches the results of the national academy evaluation and follow-up in which participants expressed the most interest in further training in the areas of curriculum and professional development, especially in the area of scientific inquiry.  Evaluation and fund development participation was also fairly strong, along with partnerships, which is a topic area added to the regional academies.  The lowest exposure was in the area of marketing, but this area was not explicitly stated in the academy request for applications (RFA).  An interesting finding is that over three-quarters of the academy participants were female. This may reflect the gender balance of the 4-H professional workforce, especially at the county 4-H agent level. But it could also have implications for potential capacity for engaging girls in 4-H Science.  Prior awareness and use of tools and resources for 4-H science revealed mixed results, with curriculum and webinars showing the greatest awareness and use. Other resources revealed low level of awareness and use, particularly fund development and urban science resources. Since participants should have learned more about these resources at the regional academy, it will be important to monitor levels of use of the resources in future evaluations.

iv

Knowledge and Skills for 4-H Science: Impact of the Regional  A positive result in the area of program evaluation is in using existing tools to evaluate 4-H Science programs. This result dovetails nicely with the results of the national science academy evaluation that showed a distinct need for ready-to-go instruments to evaluate 4-H Science programs  A significant result appears to be in the area of teaching science inquiry to facilitators of 4-H Science programs. This item shows the greatest change in reported level of ability, with 74% reporting moderate or high levels at the end of the academy. This result dovetails nicely with the results from previous academy evaluations that showed a marked need for further training in science inquiry.  The emphasis on inquiry was evident in the results in the professional development and curriculum areas. This builds nicely on the results of past academy evaluations that revealed a need for more training on inquiry. It appears that the regional academies met this request well, and served to further the infusion of science inquiry as an important aspect of 4-H Science.  There appears to be continued concern in Fund Development (especially with the toolkit) and Evaluation (especially in providing leadership for evaluation). Learning to utilize the fund development toolkit and providing leadership for evaluation were two explicitly stated outcomes for the academies, yet they appear to lag behind the rest of the outcomes. Confidence and Intention to Teach Others  The results indicate a positive personal and organizational readiness to facilitate 4-H Science  There is a general emphasis on furthering 4-H Science through curriculum and professional development.  There is less ability to facilitate 4-H Science in the fund development and evaluation areas than in curriculum and professional development. Science Liaisons Estimated Reach Beyond Regional Academies  It is interesting to note the greater reach planned to staff in the areas of infusing science and evaluation science programs. This may be an early indicator of a move away from emphasizing inquiry toward other topics, and could indicate that inquiry as a basis for 4-H Sciences is beginning to be understood and utilized more easily.  The lower levels of dissemination of information to staff related to the science plan of action, providing leadership for evaluation, and using the fund development toolkit is consistent with other findings in this report, and again perhaps indicative of the

v

specialist audience that needs to be reached about these topics. Clearly, not all staff will benefit from this information and targeted dissemination efforts should be considered. Virtual Academy  The Northeast-New England regional academy was held virtually. The evaluation of this format received mixed results, with several suggestions for improvement. Because the Year Three academies are going to be virtual, the feedback and critique from the New England Academy is presented in Appendix One of this report.

vi

Year Two: Regional 4-H Science Academies Overview
Five regional 4-H Science Academies were held during the second year of the National 4-H Science Leadership Academy project: Northeast (Mid-Atlantic), State College, PA, January 24-26. North Central-St. Louis, MO, February 7-10 West-Davis, CA, March 26-28 Southern- Huntsville, AL, April 9-12 Northeast (New England), Virtual Academy, April 10-11 Academy planners submitted a proposal in response to the Regional 4-H Science Academy RFA issued by National 4-H Council. The RFA outlined specific content, goals, and outcomes for the regional academies. Each region submitted a proposal that sufficiently met the requirements for the regional academies and received funding to plan and host the academy in their region. The stated goal of the 4-H Regional Science Academies was to: Build the capacity of state and local 4-H faculty, staff, and volunteers to offer high quality, sustainable 4-H Science programs by helping them:  Design, implement, evaluate and sustain 4-H Science programs  Develop strategies to secure funds and other resources to support 4-H Science at the state and local levels  Assist Land-grant Universities and counties with developing and refining their 4-H Science Plans of Action  Identify additional training and resources needed to support the 4-H Science Mission Mandate Furthermore, specific content was to be provided in order to meet the outcomes for the academies in the following areas: Fund Development 1. Every LGU will have a written vision and case for supporting 4-H Science financially. 2. LGUs will update their 4-H Science Plans of Action to include a revenue plan that articulates the role of government grants, fees for service (if applicable) and philanthropic support as well as a multi-year fund development plan which includes specific fundraising goals to achieve that revenue plan.

1

3. Every LGU will have a plan for identifying, recruiting and engaging volunteer champions who are passionate about 4-H and active in a variety of fund development activities in support of the sustainable expansion of 4-H Science programming within that state. Curriculum 1. Participants will increase their skills to train others on curriculum components and program quality. 2. Participants will be prepared to develop new, and revise existing curricula, incorporating identified criteria and standards. 3. Participants will be able to train others to identify how science is infused through mission mandate areas in 4-H. Evaluation 1. Participants will learn to utilize existing tools and identify appropriate evaluation methods. 2. Participants will increase their knowledge to determine when and how to do evaluation that meets their needs. 3. Participants will increase their knowledge and skills to provide leadership to and build capacity for evaluation at Land-grant Universities and at the county level. Professional Development 1. Participants will be able to utilize a train-the-trainer model to train staff and volunteers to facilitate 4-H Science programs. 2. Participants will be able to train staff and volunteers to recruit, develop, and retain volunteers to facilitate 4-H Science programs. 3. Participants will be able to train staff and volunteers to use 4-H Science professional development tools and resources. 4. Participants will be able to transfer knowledge and skills learned across all three mission mandates. Partnership Development 1. Regional academy planning teams were encouraged to offer training in partnership development. The evaluator worked directly with the lead contact person for each academy to secure contact information (names and e-mail addresses) for all academy participants. Participants were contacted via e-mail and invited to take part in an on-line post-academy evaluation. Invitations were sent immediately following the completion of each academy, with up to three follow-up
2

reminders to non-respondents over the course of 2-3 weeks. A total of 515 invitations were sent, with 427 returned of an overall return rate of 82.9%: Participant response rates for all the academies were sufficiently strong. Table 1.0 shows the number of respondents and response rates by academy.
Table 1.0 Response Rates by Academy Northeast (Mid-Atlantic) North Central South West Northeast (New-England) Total Invitations 73 228 78 100 36 515 Respondents 62 185 63 87 30 427 Response Rate 84.9% 81.1% 80.7% 87.0% 83.3% 82.9%

Participants at the regional academies were overwhelmingly female (76.8%). Sixty five respondents indicated they had attended the National 4-H Science Leadership Academy at the National 4-H Conference Center in December, 2010; 351 respondents reported they did not attend the national academy. Those who had attended the national academy were evenly distributed according to the focus track they had attended at the national level:     17 attended the curriculum focus track 17 attended the evaluation focus track 15 attended the fund development track 16 attended the professional development track

By far, the majority of participants in the regional academies were county 4-H agents (221 reported this). Additional participant roles in 4-H are presented in Figure 1.0.

3

Figure 1.0 Participant Roles in the 4-H Program
250 200 150 100 50 0 Volunteer Partner State 4-H (funder, Liaison program supporter) State 4-H Science Specialist State 4-H County 4-H County 4-H State 4-H State 4-H State 4-H Specialist Educator Educator Program Foundation Foundation (not (agent role) (non-agent Leader Staff Director science role) specific) Missing

221

63 25 33 5 31 24 36 17 4 3

* Note: Respondents could indicate more than one role; thirty-five (9%) indicated more than one role.

Respondents were provided the option of indicating their role was something other than those listed. Table 1.1 provides a summary of the “other” roles identified by respondents.

Table 1.1 Other Roles Indicated 4-H Program Coordinator Regional Agent or Specialist State Level Educator Community Worker/Educator CYFAR/Military Student County Director Retired

26 17 12 6 4 2 1 1

As stated earlier, regional academies were designed to provide professional development training for 4-H Science in: Curriculum, Evaluation, Fund Development, and Professional Development. Content related to Partnerships and Marketing was strongly encouraged. Most respondents reported experiencing content in the areas of Professional Development and Curriculum at the academies. This is consistent with the content emphasis of the academies that emphasized science inquiry in the curriculum and professional development areas. Figure 1.1 shows the number of respondents who reported participating in learning opportunities by content area.

4

Figure 1.1 Number of Participants in Academy Learning Areas
250 204 200 164 150 111 100 60 50 0 Curriculum Evaluation Fund Partnerships Professional Development Development Marketing Attended all tracks 62 101 111

* Note: Respondents could indicate participating in more than one area

One last demographic question asked participants about their pre-academy awareness and use of tools and resources to support 4-H Science (see Figure 1.2).  Approximately 250 respondents (58%) were aware of resources and tools related to curriculum and to the 4-H Science webinars. But considerably fewer had actually used these tools: Webinars- 32%; curriculum website- 44%; and curriculum pieces -47%.  Slightly less (226; 53%) were aware of the state or LGU’s science plan of action.  For professional development, 130 (30%) reported awareness of the resources, and 85 (20%) reported using them.  Just 142 (33%) reported being aware of the fund development resources, and only 37 (9%) reported using the resources.  Only 92 (22%) reported being aware of the web resource for science in urban communities, with just 32 (7%) reporting having used these resources.

5

Figure 1.2 Frequencies for Awareness and Use of 4-H Sciences Resources- Pre- Academy
300

254
250 200 150 100 50 0 National 4-H Science Webinars Fund Development Toolkit

252

255 226 201 158 130 Aware Of 92 85 32 Used

187 137 142

37

4-H Science Curriculum Website

National 4-H Science Curriculum Pieces

4-H Science in 4-H Science State or LGU 4Urban Professional H Science Plan Communities Development of Action Website Toolkit

Summary  Demographic data from the regional academies indicate that the primary audience for the academy (county 4-H agents) was reached.  Academy attendance overall appears to represent the size and involvement of 4-H Science programs for each region.  Participants report a fairly even exposure to the academy topics, with slightly more attending sessions on curriculum and professional development than the other areas. This, however, matches the results of the national academy evaluation and follow-up in which participants expressed the most interest in further training in the areas of curriculum and professional development, especially in the area of scientific inquiry.  Evaluation and fund development participation was also fairly strong, along with partnerships, which is a topic area added to the regional academies.  The lowest exposure was in the area of marketing, but this area was not explicitly stated in the academy request for applications (RFA).  An interesting finding is that over three-quarters of the academy participants were female. This may reflect the gender balance of the 4-H professional workforce, especially at the county 4-H agent level. But it could also have implications for potential capacity for engaging girls in 4-H Science.  Prior awareness and use of tools and resources for 4-H science revealed mixed results, with curriculum and webinars showing the greatest awareness and use. Other resources revealed low level of awareness and use, particularly fund development and urban science resources. Since participants should have learned more about these resources at
6

the regional academy, it will be important to monitor levels of use of the resources in future evaluations.

7

Part Two Knowledge and Skills for 4-H Science: Impact of the Regional Academy
The second part of the of the regional 4-H Science academy evaluation focused on the knowledge and skills that participants reported leaving the academies with. Respondents were asked to rate how much the academy contributed to their knowledge or skill related to important learning items contained in the academy on a nominal scale of: (1) none; (2) low; (3) moderate; and (4) high. For the purposes of this evaluation, we were interested in the number of participants that reported that the academy contributed a moderate or high level to the development of skill or knowledge about 4-H Science. However, we knew that the backgrounds of those attending the academy in regards to 4-H Science would vary considerably, so we also asked respondents to rate their level of knowledge or skill related to important learning items prior to the academy on the same nominal scale. For these questions, we were interested in the number of participants that reported “none” or a “low” level of skill or knowledge prior to the academy. In order to best understand the impact of the academies on personal skill and knowledge, both of these questions should be considered in tandem. The figures below show the results for both questions by content area. Curriculum Just under half of the respondents reported none or low levels of skill and knowledge related to 4-H Science curriculum before the academy (see Figure 2.0). This indicates that the majority of respondents had considerable experience in curriculum before attending the academy. However, the majority of respondents reported that the academy had contributed a moderate or high amount to their skill and knowledge related to 4-H Science Curriculum (se Figure 2.1). This was particularly true in:  Why inquiry is critical to 4-H Science  Program quality for 4-H Science  Curriculum components for 4-H Science

8

Figure 2.0 Percent of Participants Report “None” or “Low” Before the Academy
100.0 90.0 80.0 70.0 60.0 50.0 40.0 30.0 20.0 10.0 0.0 13.4 35.7 31.7 24.3 9.5
Revise curriculum incorporating science curriculum and standards

32.3

30.9 30.9 23.2 11.6
Identify high quality PYD science curriculum

Low 30.4 25.3 12.1 None

15.6

Curriculum Program quality Develop new components for for 4-H science science 4-H science programs curriculum

Science across Why science all 4-H mission inquiry is critical mandate areas in 4-H

Figure 2.1 Percent Reporting Academy Learning was “Moderate” or “High”
100.0 90.0 80.0 70.0 60.0 50.0 40.0 30.0 20.0 10.0 0.0
Curriculum Program Develop new Revise Identify high Science across Why science components quality for 4-H science curriculum quality PYD all 4-H mission inquiry is for 4-H science science curriculum incorporating science mandate areas critical in 4-H programs science curriculum curriculum and standards

41.0

46.4

29.8

31.4

43.5

39.8

63.1 High

41.3

Moderate 36.1 38.6 38.0 35.4 34.8 23.0

9

Evaluation About 60% of the respondents reported none or low levels of skill and knowledge related to 4-H Science program evaluation before the academy (see Figure 2.2), indicating that most of the respondents did not have strong skill for program evaluation coming into the academy . About the same percentage reported that they learned a moderate or high amount related to 4-H Science evaluation, meaning at least 40% of respondents reported learning nothing or only low levels of skill and knowledge related to evaluation (see Figure 2.3). Changes in skill and knowledge for evaluation are difficult to interpret from these results, and may be most likely due to the reality that it takes considerable time to develop good skills in program evaluation.  The most positive result is in the area of using existing tools to evaluate 4-H Science programs. This result dovetails nicely with the results of the national science academy evaluation that showed a distinct need for ready-to-go instruments to evaluate 4-H Science programs The least positive result was in providing leadership for 4-H Science evaluation. This is likely reflective of the academy audience; while 4-H agents may engage in evaluation efforts, they are not typically the ones who provide leadership for such efforts.

Figure 2.2 Percent of Participants Report “None” or “Low” Before the Academy
100.0 90.0 80.0 70.0 60.0 50.0 40.0 30.0 20.0 10.0 0.0
Use exisiting tools for Identify methods for How to evaluate a 4-H When to evaluate a 4- Provide leadership for evaluating 4-H sci. evaluating 4-H science science program H science program 4-H Science evaluation programs programs

39.8

Low 42.6 40.3 38.3 37.8 None

22.6

19.1

20.5

20.8

21.9

10

Figure 2.3 Percent Reporting Academy Learning was “Moderate” or “High”
100.0 90.0 80.0 70.0 60.0 50.0 40.0 30.0 20.0 10.0 0.0
Use exisiting tools for Identify methods for How to evaluate a 4- When to evaluate a Provide leadership evaluating 4-H sci. evaluating 4-H H science program 4-H science program for 4-H Science programs science programs evaluation

19.5

15.7

14.4

13.0

15.3

High Moderate

42.5

42.8

44.3

41.3

36.8

Professional Development Professional development appears to be an area where many respondents had considerable experience prior to the academy, with 50% or less reporting none or low levels of expertise (see Figure 2.3). This was particularly the case in the area of using train the trainer models with staff and volunteers, and for forming successful youth-adult partnerships. The ratings for postacademy levels of knowledge and skill rose considerably with over 60% reporting moderate or high levels for each skill (see Figure 2.4).  The most significant results appear to be in the area of teaching science inquiry to facilitators of 4-H Science programs. This item shows the greatest change in reported level of ability, with 74% reporting moderate or high levels at the end of the academy. This result dovetails nicely with the results from previous academy evaluations that showed a marked need for further training in science inquiry.

11

Figure 2.4 Percent of Participants Report “None” or “Low” Before the Academy
100.0 90.0 80.0 70.0 60.0 50.0 40.0 30.0 20.0 10.0 0.0

23.4 14.4
Train the trainer model for volunteers

23.7 15.3
Train the trainer model for staff

32.2 14.4

35.6 15.5

33.3 31.4 18.7 10.3
Form YAPs to support 4-H sci programs

Low None

Train staff to Train volunteers to Teach scientific recruit develop recruit develop inquiry to and retain and reatain facilitators of 4-H volunteers volunteers sci. programs

Figure 2.5 Percent Reporting Academy Learning was “Moderate” or “High”
100.0 90.0 80.0 70.0 60.0 50.0 40.0 30.0 20.0 10.0 0.0

29.5

30.6

22.9

23.6

45.8

29.2 High

41.9

37.5

42.7

39.8

31.4

44.1

Moderate

Train the trainer Train the trainer Train staff to Train volunteers Teach scientific Form YAPs to model for model for staff recruit develop to recruit develop inquiry to support 4-H sci volunteers and retain and reatain facilitators of 4-H programs volunteers volunteers sci. programs

Fund Development Fund development is the area in which respondents reported the least amount of prior expertise (see Figure 2.6). Over 50% of respondents reported no or low levels of knowledge and skill for every item, and this is particularly true in using the fund development toolkit, where 81% reported no or low levels. At the end of the academy at least 60% of respondents reported moderate to high levels of knowledge and skill for each item, with the exception of using the fund development toolkit; just 45% of respondents reported moderate or high levels for this item (see Figure 2.7).
12

These results suggest that the fund development area is one in which most 4-H agents do not have a great deal of experience. However, the reported increase in knowledge and skills in fund development as a result of the academy is fairly strong, which indicates that the academy served an important role in skill and knowledge development in the fund development area. The lack of skill to understand and use the fund development toolkit remains a concern. However, it is not entirely clear how much the use of the toolkit was emphasized during the academies. The low rating could be simple a reflection of the toolkit not being emphasized, or they could be indicative of confusion about how to use the toolkit, or it could be a reflection of the fact that most 4-H agents are not focused on fund development. Given the resources that have gone into developing the toolkit, it will be important to keep monitoring its use by 4-H Science programs.

Figure 2.6 Percent of Participants Report “None” or “Low” Before the Academy
100.0 90.0 80.0 70.0 60.0 50.0 40.0 30.0 20.0 10.0 0.0
Develop Develop Why invest in Fund Volunteer Partnershipss Identify Support fund Work with state or LGU county Plans 4-H science - development champions for 4-H prospective development volunteers to Plan of of Action for donors toolkit use for 4-H science donors for 4-H support fund Action science science development

25.8

26.4

28.5

39.0 38.5 55.7 36.4 27.0

40.2

36.4

39.2

Low None

31.1

30.6

20.6

16.5

25.4

28.6

22.1

13

Figure 2.7 Percent Reporting Academy Learning was “Moderate” or “High”
100.0 90.0 80.0 70.0 60.0 50.0 40.0 30.0 20.0 10.0 0.0
Develop Develop Why invest in Fund Volunteer Partnershipss Identify Support fund Work with state or LGU county Plans 4-H science - development champions for 4-H prospective development volunteers to Plan of of Action for donors toolkit use for 4-H science donors for 4-H support fund Action science science development

33.3 28.4 29.9 9.9 42.2 34.1 40.4 34.4 42.7 46.8 44.7 40.7 39.9 33.1 18.8 19.4 21.7 24.3
High Moderate

Summary  Overall, these results indicated that the learning at the academy was strong and positive for all of the content areas.  The emphasis on inquiry was evident in the results in the professional development and curriculum areas. This builds nicely on the results of past academy evaluations that revealed a need for more training on inquiry. It appears that the regional academies met this request well, and served to further the infusion of science inquiry as an important aspect of 4-H Science.  There appears to be continued concern in Fund Development (especially with the toolkit) and Evaluation (especially in providing leadership for evaluation). Learning to utilize the fund development toolkit and providing leadership for evaluation were two explicitly stated outcomes for the academies, yet they appear to lag behind the rest of the outcomes. It may be that there is a need to identify and match these two aspects of the academy to the right audience. Who is it at each LGU that has the capacity and right position description to further these outcomes? Given the specialized nature of these outcomes, these results may be a function of missing the audience who can best use the toolkit and/or provide leadership for evaluation. Future offerings in these areas should be targeted at those who have the capacity to achieve them.

14

Part Three Confidence and Intention to Teach Others
One of the goals of the National 4-H Science Leadership Academy is to disseminate knowledge and skills related to 4-H Science programming to professionals and volunteers throughout the National 4-H system. This goal is reflected in the structure of the academy for Year Two, in that resources were moved from the national to the regional level with the intention of reaching more people through focused regional academies. As such, it is an additional goal that participants in the regional academies will develop skills and confidence to continue to build 4-H Science programs locally by providing professional development opportunities for other 4-H professionals, volunteers, and partners at the local level. To this end, a section of the regional academy evaluation focused on participants’ confidence to use information they learned and confidence to teach the information to others. In addition, participants were asked about their intention to teach others. Participants were asked to rate their level of confidence to use each item and to teach it to others. They were also asked to rate the likelihood that they would teach each items to others as part of their 4-H Science program. The figures below show the percentage of respondents who rated each item “moderate” or “high.” Curriculum Respondents reported fairly high levels of confidence related to each of the curriculum items (see Figure 3.0). The greatest level of confidence was in understanding that inquiry is critical component of 4-H Science (90%). The lowest was in developing curriculum that incorporates science criteria and standards (69%). This is consistent with previous evaluation reports that indicate states are not planning to develop their own science curriculum, but look for curriculum that is developed by others with expertise in 4-H Science.

15

Figure 3.0 Percent of Participants Report “Moderate” or “High” Confidence to use Item (N = 309)
100.0 90.0 80.0 70.0 60.0 50.0 40.0 30.0 20.0 10.0 0.0
Curriculum Program Develop Revise Identify high How science is Why science components quality for 4-H curriculum that curriculum that quality PYD infused inquiry is a for 4-H science science incorporates incorporates science through 4-H critical programs science criteria science curriculum mission component of and standards curriculum and mandate areas 4-H science standards curriculum

47.6

46.6 29.0

32.6

45.0

38.4

58.4

High

37.2

41.4

40.4

43.0

42.1

45.3

Moderate

32.5

Respondents also reported moderate to high levels of confidence to teach curriculum items to others (see Figure 3.1). The highest level was in teaching why science inquiry is critical to 4-H Science (91%); the lowest was teaching others how to develop curriculum that incorporates science criteria and standards. The reported levels of confidence to teach others mirrors the respondents own levels of confidence in each item and supports the emerging evidence that science inquiry as a foundational part of 4-H Science is taking hold, while independent efforts to develop 4-H Science curriculum is a lesser priority for many LGUs.

16

Figure 3.1 Percent of Participants Report “Moderate” or “High” Confidenc e to Teach Others
100.0 90.0 80.0 70.0 60.0 50.0 40.0 30.0 20.0 10.0 0.0
Curriculum Program Develop Revise Identify high How science is Why science components quality for 4-H curriculum that curriculum that quality PYD infused inquiry is a for 4-H science science incorporates incorporates science through 4-H critical programs science criteria science curriculum mission component of and standards curriculum and mandate areas 4-H science standards curriculum

40.5

41.8 24.3 28.6

36.9

35.5

49.3

High

42.2

45.1

41.8

42.8

44.8

45.1

40.8

Moderate

A similar picture emerges for respondent’s plans to teach others about 4-H Science curriculum (see Figure 3.2). Most respondents intend to teach others about science inquiry for 4-H Science (90%), followed by program quality (85%) and curriculum components for 4-H Science (83%). Only 61% have intentions to teach about developing curriculum for 4-H Science. Figure 3.2 Percent of Participants Report “Moderate” or “High” Intention to Teach Others
100.0 90.0 80.0 70.0 60.0 50.0 40.0 30.0 20.0 10.0 0.0 Curriculum Program quality Develop Revise Identify high How science is Why science components for for 4-H science curriculum that curriculum that quality PYD infused through inquiry is a 4-H science programs incorporates incorporates science 4-H mission critical science criteria science curriculum mandate areas component of and standards curriculum and 4-H science standards curriculum

45.8

46.0 27.0 31.0

36.8

39.7

56.0

High

37.5

39.3

34.3

37.3

43.0

42.0

Moderate

34.1

17

In addition to determining intentions for teaching content to others, it is important to also determine intentions not to teach content to others. This provides a picture of potential problems in reaching the information dissemination goals of the regional academies. Figure 3.3 shows the percentage of respondents who indicated they did not plan to teach content to others. These ratings all fell below 15%, which is low, but still indicative that some respondents did not have plans to teach others, which could mildly impact the success of the aca demy’s goal to disseminate knowledge and skill beyond those who attended the academy. Figure 3.3 Percent of Participants Report No Intention to Teach Others
50.0 45.0 40.0 35.0 30.0 25.0 20.0 15.0 10.0 5.0 0.0 Use train-the-trainer Use train-the-trainer Train staff to recruit Train volunteers to model to train 4-H model to train staff develop and retain recruit develop and science volunteers in 4-H Science volunteers retain volunteers Teach scientific Form YAPs to inquiry to 4-H support 4-H Science science facilitators programs

9.3

11.9

12.2 9.0 6.5 8.0

Evaluation Respondents reported lower levels of confidence related to each of the program evaluation items (see Figure 3.4). The greatest level of confidence was in understanding how to use existing tools for evaluating 4-H Science programs (70%) and the lowest was providing leadership for evaluation (61%).

18

Figure 3.4 Percent of Participants Report “Moderate” or “High” Confidence to use Item
100.0 90.0 80.0 70.0 60.0 50.0 40.0 30.0 20.0 10.0 0.0 How to use exisiting How to identify Determining how to When to evaluate a 4- Provide leadership for tools for evaluating 4- appropriate methods evaluate a 4-H science H science program evaluation of 4-H H science programs for evaluating science program science programs programs

25.5

14.2

12.9

16.3

18.8
High Moderate

44.4

51.8

51.7

49.8

42.8

Respondents reported even less confidence to teach others about evaluation (see Figure 3.5). on the high side, 61% indicated confidence to teach others about using evaluation tools, while on the low side, only 55% felt confident to teach others how to identify appropriate methods for evaluating 4-H Science programs. Figure 3.5 Percent of Participants Report “Moderate” or “High” Confidence to Teach Others
100.0 90.0 80.0 70.0 60.0 50.0 40.0 30.0 20.0 10.0 0.0
How to use exisiting How to identify Determining how to When to evaluate a 4- Provide leadership for tools for evaluating 4- appropriate methods evaluate a 4-H science H science program evaluation of 4-H H science programs for evaluating science program science programs programs

18.9

11.4

10.8

12.8

15.8

High Moderate

42.1

43.1

47.8

47.1

39.3

19

Despite the lower levels of reported confidence related to evaluation and teaching to others, over 50% reported a moderate or high intention to teach these concepts to others (see Figure 3.6). Figure 3.6 Percent of Participants Report “Moderate” or “High” Intention to Teach Others
100.0 90.0 80.0 70.0 60.0 50.0 40.0 30.0 20.0 10.0 0.0
How to use exisiting How to identify Determining how to When to evaluate a Provide leadership tools for evaluating 4- appropriate methods evaluate a 4-H 4-H science program for evaluation of 4-H H science programs for evaluating science science program science programs programs

24.8

16.3

16.0

17.0

19.0

High Moderate

39.1

43.2

43.2

40.1

36.6

More respondents reported that they did not plan to teach others in the area of program evaluation. As Figure 3.7 shows, the percentage ranged from 10 % for “how to use existing tools” to 17.6% for “providing leadership for program evaluation. These results could reflect the fact that program evaluation is a complex topic that takes time to master, as well as that most county agents are not typically involved in teaching evaluation to others. The bright note in these results is that teaching others to use existing tools for 4-H Science programs appears to be the topic that will be taught the most to others. This supports the findings of previous evaluations that indicated having access to ready tools for evaluation should be a priority.

20

Figure 3.7 Percent of Participants Report No Intention to Teach Others
50.0 45.0 40.0 35.0 30.0 25.0 20.0 15.0 10.0 5.0 0.0 How to use exisiting tools How to identify for evaluating 4-H science appropriate methods for programs evaluating science programs Determining how to evaluate a 4-H science program When to evaluate a 4-H Provide leadership for science program evaluation of 4-H science programs

15.3 10.1

15.0

15.6

17.6

Professional Development Over 70% of respondents reported moderate or high confidence levels to use and teach items related to professional development (see Figures 3.8 and 3.9). Intentions to teach others dropped off slightly, especially around training staff and volunteers to identify, recruit and retain volunteers for 4-H Science. Even so, over 65% reported moderate or high intentions to teach others about all professional development items (see Figure 3.10). Figure 3.8 Percent of Participants Report “Moderate” or “High” Confidence to use Item
100.0 90.0 80.0 70.0 60.0 50.0 40.0 30.0 20.0 10.0 0.0
Use train-theUse train-thetrainer model to trainer model to train 4-H science train staff in 4-H volunteers Science Train staff to Train volunteers Teach scientific Form YAPs to recruit develop to recruit develop inquiry to 4-H support 4-H and retain and retain science Science programs volunteers volunteers facilitators

38.1

37.0

25.3

22.6

38.8

38.6 High

41.5

38.4

51.0

50.8

43.1

44.6

Moderate

21

Figure 3.9 Percent of Participants Report “Moderate” or “High” Confidence to Teach Others
100.0 90.0 80.0 70.0 60.0 50.0 40.0 30.0 20.0 10.0 0.0
Use train-theUse train-thetrainer model to trainer model to train 4-H science train staff in 4-H volunteers Science Train staff to Train volunteers Teach scientific Form YAPs to recruit develop to recruit develop inquiry to 4-H support 4-H and retain and retain science Science programs volunteers volunteers facilitators

36.1

34.7

21.8

20.1

32.8

33.3 High

43.9

39.8

51.9

51.0

46.3

Moderate 44.6

Figure 3.10 Percent of Participants Report “Moderate” or “High” Intention to Teach Others
100.0 90.0 80.0 70.0 60.0 50.0 40.0 30.0 20.0 10.0 0.0
Use train-theUse train-thetrainer model to trainer model to train 4-H science train staff in 4-H volunteers Science Train staff to Train volunteers Teach scientific Form YAPs to recruit develop to recruit develop inquiry to 4-H support 4-H and retain and retain science Science programs volunteers volunteers facilitators

39.5

37.4

27.6

22.2

36.1

36.3 High Moderate

35.1

31.5

39.5

43.2

43.3

39.4

An analysis of the respondents who reported no intention to teach others about curriculum topics revealed relatively low percentages ranging from 6.5% for “teaching scientific inquiry” to 12.2% for “ training staff to recruit volunteers.” These results are similar to those found for the curriculum area and are indicative of the general emphasis placed on furthering 4-H Science through curriculum and professional development training.

22

Figure 3.11 Percent of Participants Report No Intention to Teach Others
50.0 45.0 40.0 35.0 30.0 25.0 20.0 15.0 10.0 5.0 0.0 Use train-the-trainer Use train-the-trainer Train staff to recruit Train volunteers to model to train 4-H model to train staff develop and retain recruit develop and science volunteers in 4-H Science volunteers retain volunteers Teach scientific Form YAPs to inquiry to 4-H support 4-H Science science facilitators programs

9.3

11.9

12.2 9.0 6.5 8.0

Fund Development Things are a bit less positive in the fund development area, with respondents reporting considerably less confidence to use some of the fund development content (see Figure 3.12). This is particularly true for confidence to use the fund development toolkit, with only 48% reporting moderate or high confidence. Figure 3.12 Percent of Participants Report “Moderate” or “High” Confidence to use Item (N=316)
100.0 90.0 80.0 70.0 60.0 50.0 40.0 30.0 20.0 10.0 0.0

25.9

28.1

19.6 11.3 13.1 45.0 52.8

29.6

13.9 51.3

15.7 46.3

16.5 48.4 High Moderate

47.8

46.4

36.3

51.9

Develop a Develop Make case Utilize fund Identify Develop Identify Work with Work with state or LGU county Plans for investing development recruit, partnerships prospective others to volunteers to Plan of of Action in 4-H toolkit engage for 4-H donors support fund support fund Action science for volunteer science development development donors champions for 4-H for 4-H science science

23

Confidence to teach the fund development to others drops off even more (see Figure 3.13), for each of the items. The one that remains strongest is “developing partnerships for 4 -H Science” where 78% of respondents indicated moderate or high confidence to teach to others. A similar pattern emerges for intention to teach others (see Figure 3.14). Figure 3.13 Percent of Participants Report “Moderate” or “High” Confidence to Teach Others
100.0 90.0 80.0 70.0 60.0 50.0 40.0 30.0 20.0 10.0 0.0 Develop a Develop Make case Utilize fund Identify Develop Identify Work with Work with state or LGU county Plans for investing development recruit, partnerships prospective others to volunteers to Plan of of Action in 4-H toolkit engage for 4-H donors support fund support fund Action science for volunteer science development development donors champions for 4-H for 4-H science science

21.8

26.9

25.0 16.3 7.5
High

9.7

11.5

13.5

12.7

44.2

45.5

46.8

35.7

46.0

52.6

47.0

43.5

45.9

Moderate

24

Figure 3.14 Percent of Participants Report “Moderate” or “High” Intention to Teach Others
100.0 90.0 80.0 70.0 60.0 50.0 40.0 30.0 20.0 10.0 0.0 Develop a Develop Make case Utilize fund Identify Develop Identify Work with Work with state or LGU county Plans for investing development recruit, partnerships prospective others to volunteers to Plan of of Action in 4-H toolkit engage for 4-H donors support fund support fund Action science for volunteer science development development donors champions for 4-H for 4-H science science

31.2 27.2 33.8 24.2 19.1 13.2 41.4 37.8 46.8 41.6 40.8 43.0 18.8 19.3 18.5
High Moderate

35.1

35.7

31.4

The percentage of respondents reporting that they have no intention to teach others, however, is highest for the fund development area content (see Figure 3.15). This is particularly true with teaching others how to utilize the toolkit (22.8%), developing the state or LGU plan of action (15.1%) and working with others to support fun development for 4-H Science (15%). On a bright note, only 7.1 percent indicated no plans to teach about developing partnerships for 4-H Science.

25

Figure 3.15 Percent of Participants Report No Intention to Teach Others
50.0 45.0 40.0 35.0 30.0 25.0 20.0 15.0 10.0 5.0 0.0 Develop a Develop Make case for Utilize fund state or LGU county Plans investing in 4- development Plan of Action of Action H science for toolkit donors Identify Develop recruit, partnerships engage for 4-H volunteer science champions for 4-H science Identify Work with Work with prospective others to volunteers to donors support fund support fund development development for 4-H science

22.8 15.1 11.7 11.4 14.8 7.1 13.0 15.0 12.6

Readiness to Facilitate 4-H Science at Participating Land Grant Universities
One last question in this section related to dissemination of knowledge asked participants about their personal and organizational readiness to facilitate 4-H Science in their state. Participants were asked to rate their level of agreement with several statements related to personal and organizational readiness to facilitate 4-H Science programs at their LGU (N = 330).
  

92.4% agreed or strongly agreed that the team attending the regional academy from their LGU is ready to facilitate 4-H Science programs 91.8% agreed or strongly agreed that there is a positive environment at their LGU for 4-H
Science

92.5% agreed or strongly agreed that they have the skills needed to develop partnerships for 4-H Science

Figure 3.16 shows the percentage of respondents who “agreed” or “strongly agreed” with each item.

26

Figure 3.16 Percentage of Respondent Agreement for Personal and Organizational Readiness
100.0 90.0 80.0 70.0 60.0 50.0 40.0 30.0 20.0 10.0 0.0

29.7

33.5

32.6

32.0

46.2

26.4

21.3

61.8

54.9

59.8

54.9

45.6

51.5

59.8

Strongly Agree Agree

Skills to Ready to develop facilitate 4-H partnerships for Science 4-H Science

Group LGU is Positive Adequate Prepared to attending organizationally environment at support at LGU form youthacademy is ready to LGU for 4-H for 4-H Science adult ready to support 4-H Science partnerships to facilitate 4-H Science support 4-H Science Science

Summary  The results indicate a positive personal and organizational readiness to facilitate 4-H Science  There is a general emphasis on furthering 4-H Science through curriculum and professional development.  There is less ability to facilitate 4-H Science in the fund development and evaluation areas than in curriculum and professional development.

27

Part Four Science Liaisons: Estimated Reach Beyond Regional Academies
Estimates of Knowledge and Skill Dissemination Thirty-three participants in the regional science academies indicated they are the 4-H Science Liaison for their state or LGU. Twenty-three of the 33 (70.0%) responded to questions asked of the liaisons only. These questions asked liaisons to estimate the total number of people that they plan to reach for 4-H Science using the tools and resources provided as part of the academy. Liaisons were asked to report this information for their whole state or LGU, not just for them personally. High and low parameter estimates were calculated to create a range of estimated reach for disseminating content area information to each audience. The Table 4.0 presents the estimated ranges for reaching others with content. The estimates are organized by audience (staff, volunteers, and partners). Cells highlighted in GREEN indicate the areas where the most reach is planned; cells highlighted in RED indicate the areas where the least reach is planned. Table 4.0 Liaison Estimates of Knowledge and Skill Dissemination Topic
Developing State or LGUs science Plan of Action Science infused throughout three 4-H mission mandates Evaluating 4-H Science Programs Teaching scientific inquiry to facilitators of 4-H Science Making the case for investing in 4-H Science Providing leadership evaluating 4-H science programs Using the train-the-trainer model to train others Developing partnerships for 4-H Science Developing curriculum that incorporates inquiry, science criteria and standards Revising curriculum to incorporate inquiry, criteria and standards Utilizing the fund development toolkit Engaging champions for 4-H Science fund development

Staff 233-545 628-1055 628-1050 568-985 222-520 171-430 426-760 407-755 316-640 496-870 220-485 259-525

Volunteers 383-650 1664-2151 1426-1985 2046-2221 945-1360 712-1060 1086-1585 1077-1545 767-1125 1248-1815 820-1135 800-1205

Partners 22-146 67-213 55-177 73-235 85-271 19-113 567-713 551-729 28-116 551-677 27-105 209-410

28

Staff
  The greatest planned reach for staff is in the areas of 1) teaching how science is infused throughout the mission mandate areas; and 2) evaluating science programs. The least amount of planned reach is in the areas of 1) developing the science plan of action; 2) providing leadership for evaluating 4-H Science programs; and 3 utilizing the fund development toolkit.

Volunteers  The greatest planned reach for volunteers is in the areas of 1) infusing science throughout the three mission mandate areas; and 2) teaching scientific inquiry to facilitators of 4-H Science programs.  The least amount of planned reach for volunteers is in developing the state of LGU science plan of action. Partners  The areas with the greatest planned reach for partners are: 1) using the train-the-trainer model to train others; 2) developing partnerships for 4-H Science; and 3) revising curriculum to incorporate inquiry, criteria and standards.

Summary While it is important to note that the figures presented in Table 4.0 are only estimates and only from twenty-three state Science Liaisons, they do paint an interesting and consistent picture of the potential dissemination reach of skills and knowledge from the regional academies.  It is interesting to note the greater reach planned to staff in the areas of infusing science and evaluation science programs. This may be an early indicator of a move away from emphasizing inquiry toward other topics, and could indicate that inquiry as a basis for 4H Sciences is beginning to be understood and utilized more easily.  The lower levels of dissemination of information to staff related to the science plan of action, providing leadership for evaluation, and using the fund development toolkit is consistent with earlier findings in this report, and again perhaps indicative of the specialist audience that needs to be reached about these topics. Clearly, not all staff will benefit from this information and targeted dissemination efforts should be considered. Additional Resources Garnered for 4-H Science Liaisons were also asked about new partnerships, gifts, grants, and other donations that have been leveraged to facilitate 4-H Science in their state. Liaisons reported the following funding secured for 4-H Science:  Gifts: $159,000.00 (6 liaisons reporting)
29

  

Grants: $2,007,900 (14 liaisons reporting) Donations: $102,000 (2 liaisons reporting) In-kind estimates: $135,390 (5 liaisons reporting)

In addition to financial resources, Table 4.1 lists the new partnerships developed for 4-H Science by LGUs. Table 4.1 New Partnerships formed by LGUs to Facilitate 4-H Science Partnerships
       Robotics support through National 4-H Council We are working with libraries and a state level contact for libraries - trying to increase 4-H STEM programs in libraries and connect to literacy. We've applied for Smith-Lever funding for a project to develop materials for a 4-H summer STEM/literacy library program. We are increasing efforts to connect with campus partners in departments and research centers. We have secured funding for our statewide STEM Program Work Team to hold a spring professional development/STEM Plan development retreat. OJJDP 4-H Tech Wizards Walmart - HealthJam and Summer Nutrition Programs JCP Robotics USB Biotech ADM Grant to Support Think Green Environmental Program Received grants from United Soybean Board for biotech, TecXite NSF subgrant for engineering, Gear-tech-21 NSF subgrants for robotics and Geospatial, and grants from MFA and JC Penney for robotics. Several corporate partners: Best Buy, 3M, CASE Int. Corporate Partnerships include Lowe's & Walmart. Institutional partnerships include Oklahoma State University and University of Oklahoma. Working to establish a relationship with other higher education institutions. - LGU departments (i.e. Physics, Education, Engineering, etc.) - 4-H Foundation partnering to find funding for 4-H Science programs - LGU Engineering Alumni to partner with county educators - Association with other state regional universities. Tennessee Geographic Information Council is support of a GIS 4-H program. Foundation support from within the Texas 4-H Program and from outside foundation sources have been very generous in the 4-H Science area. We have been using our extension funds, resources we obtained from other mini-grants, and collaborating with LGUs in the state for science resources (staff training and resource materials and facilities) Partnership with a public utility to support the NYSD Industry partners to support robotics program Grants to support science programs California Afterschool Network - Professional development and 4-H curriculum dissemination. California Science Teachers' Association - 4-H workshops at their conferences Grants from National 4-H Council FairPoint Communications - funds for equipment and volunteer training; UVM Extension 30

  

      

 

AmeriCorps & VISTA support funding; State 4-H Foundation equipment grants; The University has created a new position that will focus entirely on 4-H Science. They have also shifted and identified 4-H Science as a focus for all Extension staff. JCPenney Foundation (4-H Robotics Grant) ESRI -GPS grant Funding from private source for development of junior master gardener program

Finally, Liaisons were asked to describe other developments in their 4-H Science program that are direct results of the national and or regional 4-H science academy programs. These developments are presented in Table 4.2

31

Table 4.2 Developments in 4-H Science as a Result of the National and Regional Science Academy Programs.
           Our staff are making the "subtle shifts" to inquiry based learning. Strategic planning at the state level Primarily work on our state STEM plan as a result of the National Academy. As a result of the Regional Academy we will be doing more revisions to our state plan and holding a Program Work Team retreat and a State Academy for county Educators and volunteers. Simply organization and unifying state objectives and delivery methods. Also build networks and validate our work. Very beneficial! Extension administration support of funds for staff professional development in 4-H Science (60 staff attended regional academy) Determined to train all staff in the inquiry process in the coming year. To figure out how to incorporate the inquiry process into curriculum and teaching lessons for state wide specialists. Hosted NC regional 4-H science academy with approximately 55 Missouri delegates. Networking with other LGU's and picking up on some of the resources has been a direct result of these academy efforts. Marketing logos, templates, - curriculum templates, new curricula related to science/STEM Shared programs, curricula, etc. - Developed network of educators, support each other Discussion about a 4-H STEM Specialist and this person's involvement on a campus-wide outreach coordinators' committee. As a result of participating in the National 4-H Science Academy, we have developed interest in increasing youth participation in science and agricultural science related careers. We have also submitted a proposal to USDA/NIFA to build our extension staff capability to organize youth in schools and communities with limited resources to develop interest in science. We are utilizing the resources provided to us during the 4-H Science Academy to conduct effective assessment of our science program State-wide awareness of 4-H Science Mission Mandate Discussion on how to better organize 4-H programs and develop 4-H Project info sheets. more marketing to promote and recognize science efforts Development of the Science Rich 4-H Project Handbooks Resources leveraged in new state 4-H Science resource library and 17 new state 4-H Science learning kits, new 4-H Science webpage, new state 4-H Science volunteer recruitment survey, greater Extension investment in teacher & staff training, development of state 4-H Science Post-programming technical assistance & support to extend 4-H Science learning. Increase in trainings offered to staff and volunteers around 4-H science. Have completed 4-H Science training with at least 30 staff and over 150 volunteers. Bigger focus on evaluating programs and efforts. More use of Train the Trainer with volunteers Larger awareness and efforts with fundraising

       

32

Part Five Narrative Question Content Analysis
Participants were asked three narrative questions: 1) list three things they planned to do related to 4-H Science in the next three months; 2) list the main thing related to 4-H science that they need additional support with the most; and 3) list three ideas/needs for postacademy training opportunities. A content analysis of the 1663 responses to these questions was conducted using the qualitative analysis software MAXQDA. For each question, comments were categorized into common themes. The categories for each question, as well as the percent in which each category was mentioned among the comments, are listed below (see Tables 5.0 – 5.2). Some comments contained two or more themes and so one comment could have several categories attached to it. The most frequently mentioned themes for each question are described in slightly more detail with examples of typical comments under these themes. Three Plans to Implement in the Next Three Months After analyzing the 889 comments, 19 different themes were decided upon that represented the responses related to plans for the next three months (see Table 5.0). The most common responses related to the following themes:  Planning, leading, developing, or assisting a science program (16.3% of responses)

This included work on urban science programs, geospatial programs, science Saturdays, science spin clubs, science camps, filmmaking, after-school programming, Youth Science Research Exhibition, food science, garden projects, and science programs in general.

Recruiting, training, or informing volunteers and staff on science (14.4% of responses)

This included teaching regional, state, and local volunteers, sharing with staff what was learned at academy, training volunteers and staff to incorporate science in programs and lessons, recruiting volunteers to facilitate science programs, training afterschool staff and agents, encouraging state teams to develop trainings for staff and volunteers, designing a volunteer training, and leading workshops.

Inquiry based learning (12.7% of responses)

This included learning more about inquiry based learning, training others on inquiry based learning, adding inquiry based learning to existing programs, using the inquiry method with all 4-H projects, offering support with the inquiry based approach, providing hands-on inquiry based opportunities, revising trainings to meet inquiry based standards, and adjusting lesson plans to include an inquiry based approach. 33

Involving more science in existing programs (7.9% of responses)

This included involving more science in camps, conferences, retreats, current projects (i.e. animals, clothing, foods, shooting sports), workshops, judging events, afterschool programs, fairs, recognizing science in all projects, and supporting volunteers in bringing out the science in their project.

Curriculum (7.4% of responses)

This included developing 4-H science curriculum, using curriculum ideas to teach leaders, enhancing curriculum, researching curricula, reviewing curricula to assure science model, adding inquiry based approach to curriculum, informing others of curriculum to use, accessing websites for curriculum, using the national curriculum development template, conducting webinars on 4-H Science curriculum, and borrowing other states’ curriculum.

Table 5.0 Plans to Implement in the Next Three Months
Percent Mentioned 16.3% 14.4% 12.7% 7.9% 7.4% 6.3% 6.2% 6.0% 5.5% 4.6% 2.6% 2.2% 1.8% 1.6% 0.7% 0.5% 0.4% 0.2% 2.7%

Planning, leading, developing, or assisting science programs Recruiting, training, informing volunteers and staff on science Inquiry based learning Involving more science in existing programs Curriculum Securing partnerships Fund development (i.e. use fund development toolkit, identify opportunities, seek donors and grants) State or county science plan of action (i.e. review, evaluate, or develop plan) Evaluation of science programs (i.e. use evaluation tools, plan an evaluation, improve evaluation) Robotics (i.e. facilitate a robotics club, offer robotics workshops) Using webinars, on-line resources, and national website for information on science Networking with colleagues Obtaining and using science resources Promotion of science (i.e. make 4-H science more visible, include science in monthly newsletter, promote science to youth) Obtaining more information on what was learned at academy Academy did not provide what was needed (i.e. no new information, too much lecture) Recruiting and training youth Train others in Positive Youth Development Other (i.e. set goals, write articles for volunteers, survey needs in community, set dates for meetings, write program descriptions)

34

Response Themes for Additional Support Needs After analyzing the 268 comments, 18 different themes were decided upon that represented the responses related to additional support needed (see Table 5.1). The most common themes are listed below. It is interesting to note that Fund Development and Evaluation are the areas in which most additional support is requested. This is a marked change from the Year One evaluation and follow-up that indicated the greatest need was in the area of scientific inquiry, and perhaps yet another indicator that the knowledge and skill needs for facilitating 4-H Science are shifting. Themes:  More funds and fund development (19.7% of responses)
This included funds for programs, staff, curriculum, and supplies, funding on county and state levels, funding opportunities and training, and finding donors.

Evaluation support and tools (11.2% of responses)

This included evaluations to show impact to partners and donors, how to use existing evaluation tools, ready-to-use standard evaluations, activities that include evaluations, webinars on evaluation, and understanding evaluation at different levels of program development and delivery.

More volunteers, staff, and manpower (9.7% of responses)

This included finding, recruiting, maintaining, and training volunteers, more staff to run programs, promoting volunteer opportunities, staff with evaluation experience, and volunteers with commitment.

Materials, resources, and equipment (9.1% of responses)

This included new equipment and computers, access to science materials and 4-H science resources, easy and quick science resources, storage space, promotional materials, volunteer training resources, tool-kits, one-page science activities, and curriculum.

Training related to science inquiry and hands-on science activities (7.9% of responses)

This included more in-depth understanding of inquiry learning, inquiry based curriculum examples, hands-on learning programs to share, tools to help volunteer and staff use inquiry-based learning, applying inquiry based learning to areas that do not “appear” to be science related, teaching traditional leaders to incorporate inquiry into existing programs, and hands-on demonstrations.

35

Table 5.1 Additional Support Needed
Percent Mentioned 19.7% 11.2% 9.7% 9.1% 7.9% 6.4% 5.5% 4.8% 4.2% 3.3% 3.0% 2.7% 1.8% 1.5% 1.5% 0.9% 0.9% 5.8%

More funds and fund development Evaluation support and tools More volunteers, staff, and manpower Materials, resources, and equipment Training related to science inquiry and hands-on science Obtaining and developing curriculum and content (i.e. easy to use curriculum, how to use curriculum) Training volunteers and staff (i.e. state training for volunteers, training for staff to identify science in programs) Time Institutional support, organization, and planning Utilizing science in already existing programs More information, workshops, or conferences on science Partnerships (i.e. establishing science partners, partnering with colleges and universities, adult-youth partnerships) Promotion of science and programs Creating a plan of action Support of curriculum revision Access to materials at academy Applying what was learned at academy Other (i.e. update state website, knowledge on subject matter, consistency in implementation, robotics)

Response Themes for Three Ideas or Needs for Post-Academy Training Opportunities After analyzing the 506 comments, 24 different themes were decided upon that represented the responses related to additional training opportunities (see Table 5.2). The most common responses related to the following themes:  Recruiting and training of volunteers and staff (10.5% of responses)

This included recruiting more committed volunteers, training volunteers, staff, and state leaders in science, training on science needs, state and local training, face-to-face trainings, and training that includes a shadowing component.

Curriculum (9.8% of responses)

This included training on national 4-H curriculum, curriculum that applies to different ages, easy-touse curriculum suggestions, adapting curriculum to inquiry-based, developing curriculum, hands-on 36

training using curriculum, how to use curriculum, online trainings for curriculum, and a place to receive feedback on curriculum.

Sharing capabilities for resources, information, and success stories (9.4% of responses)

This included resource site with links to other science 4-H web sites, blogs or eNewsletters that help with sharing best practices, online community of academy attendees, regional and statewide sharing of ideas, weekly tips on science learning, sharing of states marketing tools, games, and skills, website listing key resources covered, contacts from academy, and shared materials.

Fund development (8.3% of responses)

This included finding potential donors, funding support from NSF and/or USDA, funding sources and opportunities, follow-up on fund toolkit, examples of a 4-H Science programs utilizing multiple types of funding mechanisms, and training staff on fundraising.

Evaluation (7.9% of responses)

This included how to use existing tools for evaluation, designing impact evaluations, evaluation resources and strategies, how and when to evaluate programs, simple evaluations to go with activities, and practical evaluation help.

37

Table 5.2 Three Ideas or Needs for Post-Academy Training Opportunities
Percent Mentioned 10.5% 9.8% 9.4% 8.3% 7.9% 7.2% 7.0% 4.5% 4.3% 4.0% 3.8% 2.8% 2.1% 1.9% 1.9% 1.7% 1.5% 1.3% 0.9% 0.9% 0.6% 0.6% 0.6% 6.6%

Recruiting and training of volunteers and staff Curriculum Sharing capabilities for resources, information, and success stories Fund development Evaluation Inquiry-based learning (i.e. hands on inquiry training, mentor in inquiry training, integrating inquiry into programs) Partnerships (i.e. developing and recruiting) Promoting and marketing science and information learned Specific science programs (i.e. Robotics and Gear teach 21) Webinars on various topics (i.e. curriculum, volunteers, funding) Informal science, integrating science into existing projects Hands-on science training Developing and implementing state/county plans Grant writing and grant opportunities Specific topics (i.e. PYD, citizenship and science, youth leadership, team building, learning styles) Another academy Train-the-trainer Technology (i.e. use of ScienceHub in my4-H.org, social media training) Resources (i.e. obtaining, training, and implementing) Follow-up on progress made by programs after the academy Time Professional development Science connected with other mandates Other (i.e. tour science museums, more handouts, individual follow-up, national science requirements)

38

Appendix 1: Feedback from New England on Effectiveness of Virtual Academy
Table 13.0 Northeast New England additional questions
How effective was the virtual format of the academy? Not effective at all Somewhat effective Effective Very effective Missing Do you think virtual Yes training is as effective as No face to face training? Missing Which type of training do Virtual you prefer? Face to face, on site No Preference Missing Why do you prefer virtual Reduced time away from office training? Reduced Travel costs More effective use of my time Easier to focus on content that is presented
Why do you prefer face to face training?

Frequency 5 13 4 1 7 7 15 8 2 16 5 7 2 2 2 2 3 13

% 16.7 43.3 13.3 3.3 23.3 23.3 50.0 26.7 6.7 53.3 16.7 23.3 100.0 100.0 100.0 100.0 18.75* 81.25*

More effective use of my time Easier to focus on content that is presented I learn more this way

10 Provides network opportunities 10
*Percentage was calculated as: the number who responded/the number who said face to face training was preferred

62.5* 62.5*

Open Ended Questions: Comments about the Virtual Training format:  1. Test the technology FIRST. It was so frustrating to have presenters unfamiliar with the format they were using. There was no forethought about how to present to multiple locations. 2. There was the ability to do hands on pieces- however this was not done. I think if the planning team had spent more time thinking though ways to make this day more interactive the training would have been much better. 39

 

    

      

At times the slide(s) on the screen were too small to read! I think we could just have easily taken part in this training from our own desks. It was difficult to sit for long periods listening, and reading - incorporating more time to interact with the website tools would have been helpful. As a region and state we rarely have time for discussion and planning. It could have been more effective if it was better organized. When we broke up into groups the directions were consistently vague and hard to follow. It was awful, totally boring and in my opinion misrepresented. Lapsed periods of engagement due to impersonal nature of this method of delivery. Too many technical difficulties leading to distraction. Not what I expected - way, way too much sitting and getting lectured to. Don't need someone to read a handout to me. Some of the presentations were helpful; others had technical problems that limited effectiveness. e.g. some slides had too much content, making them hard to read. Time was wasted, even in effective presentations, when presenters had trouble loading their power points or websites. URLs were not clearly displayed. Small group activities lacked sufficient direction and structure. The "hybrid" model of listening to a presenter and then spending time working with or discussing the information in our room was very effective. The slow pace of a webinar and the slow conversation make is somewhat painful to sit through, but the savings in travel time is worth it. Also, presenters could present less information and give people more time to explore what they want to/need to know about that info. The power points got tiresome after a while and the day could have been better organized with onsite personnel beforehand. The technology was difficult, but in lieu of long distance travel, I thought it was OK. I think the host could have had us do more experiential activities in between webinars Too long Need active activities/break outs. Several things would have led well to engage in role-play like approaching donors.... I would have liked to see the inquiry piece done as an activity and possible to have reviewed the Osborne piece ahead of time in prep for a role play. Too many long, involved PowerPoint slides. No opportunity for people to discuss or process between sessions. People wanted more hands-on opportunities to develop skills being taught within each session Went well for the first time. Wish there was more hands-on opportunities. Most of the sessions could have been accomplished in a webinar format. When presenters showing content from websites, very difficult to read and follow when in projected format.

Comments about Face to Face Training vs. Virtual Training:  As effective if planned to hold viewer's interest with interactive discussions, hands on activities and better post activity sharing.  Virtual training can be as useful and successful if implemented in a way that makes it as engaging as being at a workshop in person. I do not however feel that this particular training achieved this goal. Why do you prefer Face to Face Training? 40

      

Allows for movement, interaction with others, easier to relate to the presenter, and potentially less PowerPoint presentations. Body language is such an important part of presentation. We teach kids to public speak, and then we are not even using it as a training method for ourselves? We worry about kids not being able to co Content provided is usually presented in a more engaging manner. If a face to face meeting is just a power point, then it is not preferable. Gathering in regional groups as we did for the regional virtual training provides some of the networking opportunities. However, I don't think we got to know each other as well as in a face to face Hands on activities bring the true essence of the program to the participants I prefer person al contact w/ people I am working with. I learn better when I'm doing than when I'm listening. Our small states benefits from the networking to see what our colleagues are also doing with limited staff/resources.

41

Master your semester with Scribd & The New York Times

Special offer for students: Only $4.99/month.

Master your semester with Scribd & The New York Times

Cancel anytime.