Discussion with Lorna Fox from Gloucestershire Wildlife Trust about the importance of systematic evaluation. This discussion was framed by three papers:
The state of evaluation in environmental education
Most educators reported doing some form of evaluation at least weekly. The most frequent goals were to: Improve quality, Report to funders, See if work meets needs of participants, and to see if they were achieving project goals
Although many environmental educators report that they are evaluating regularly, this is mainly monitoring numbers of participants or assessing enjoyment with observations of participants during lessons. The most frequently used methods were indeed questionnaires and feedback forms, although no one mentioned using them in a pre- and post-test design. Other methods frequently reported included games, informal discussions, and letters from participants.
Professional evaluators or researchers would not consider many of the methods used for data collection appropriate. However more robust methods for collecting data and longer-term evaluations are often impractical for environmental educators to conduct themselves.
Using numbers and observations has limitations
Numbers of participants and repeat bookings does not directly indicate that objectives like increasing connection to nature or learning concepts are being met. Repeat visits may come from proximity to school, cost, or teacher familiarity with the site and feeling more comfortable leading a trip to a familiar place. None of these factors reflect the effectiveness of a guided session.
Evaluating student engagement using observations made during the session is a poor indicator of learning. Students might mask their level of engagement or fun for any number of reasons. Concern about perceptions of peer/friend groups, learned coping mechanisms to avoid or attract teacher attention etc.
Multiple routes towards and measures of success
Different organizations can have very different goals.
- Increased knowledge/environmental literacy
- Reconnection with nature
- Willingness to take action on environmental issues
- Increased physical activity
- Confidence in a practical skill
Educators and organizations should remember that there are many ways to achieve any of these goals. Quieter, contemplative experiences can be just as lasting as playful, energetic sessions. Worrying or distressing information about problems can spark willingness to take action as much as beautiful or fascinating encounters with nature. Offering a range of experiences increases the chances that participants with find an experience which speaks to them.
Identifying what specific goal is the main focus of a session is very important when considering evaluation tools. Think about what you are evaluating, and make sure it actually measures your desired outcome. For example, participants may really enjoy playing a chasing game intended to teach them about food chains. Asking students if they enjoyed the game does not tell you if the game was successful in teaching anything about the topic.
Improving use of evaluation
-
Identify the main goal of a session and use an appropriate measurement tool.
- If you want participants to feel connected to nature, use a nature connectedness survey tool.
-
Try and get pre and post-session data.
- Ask the class to do a survey beforehand; the day before, upon arrival etc.
- Plan in time at the end of a session for the class to do the post session survey.
-
Identify opportunities to have a control group.
- Look for bookings where part a visit from a year group is split across more than one day.
- Aim for all classes to complete 3 surveys: before the visit, after the visit, after an in-school lesson on the same topic.
-
Be consistent but realistic in use of evaluation.
- Some evaluation is better than none
- Find a target frequency which will work for your capacity. (Annually, quarterly, weekly etc.)
-
If an evaluation tool isn’t working for you, stop using it!
- There are many survey tools which have been developed, don’t be afraid to switch.
- Modify wording to suit your context, particularly if the results are only going to be used internally.
- Shorten survey tools if you are concerned about time. It will be more difficult to compare results with more formal research, but can still provide you with valuable information.
-
Use external evaluators or connect with research institutions / teacher training programmes
