We asked a range of literature and literacy organisations whether, why and how they measure their success. Some, such as smaller or volunteer organisations, do no formal evaluation at all. Others adopt a range of methods, dictated partly by the nature of their projects and partly by the demands of existing or future funders.
We found half a dozen main evaluation frameworks, of increasing rigour and scope:
- Commercial: sales, quality (reviews, prizes)
- Arts: artistic quality, participation, satisfaction, audience development
- Arts education: hard and soft measures of achievement, enjoyment and personal development
- Heritage: participation, satisfaction, cognitive gain. The Museums and Libraries Association has created an Outcomes framework which includes General Learning Outcomes and Generic Social Outcomes.
- Education: educational research often involves systematic measures of attitude and attainment and may include control groups and longitudinal follow-up to check for lasting changes. At the scientific end, some researchers even observe brain activity and changes.
- Charity: Some funders seek to compare charities’ impacts by measuring social return on investment, relating hard and soft measures of changes to social outcomes to the costs of making those changes.
- Economic impact: Similarly some arts and heritage organisations are asked to justify themselves by demonstrating a wider economic impact for their work (eg to tourism, creative economy).
Many of the people we spoke to felt that evaluation should be kept in perspective. Sometimes it is not desirable to measure an outcome, as the act of measurement can spoil the experience, particularly for children, who may not be willing or able to articulate their views. Sometimes it is difficult to get an accurate response. Children have a tendency to say what they think adults want to hear. For this reason, evaluations often combine comments from children, their gatekeepers, any artists involved and sometimes independent expert peers. Sometimes it is unrealistic to expect a measurable change from a small experience. And if a change is detected immediately will it ‘wash away’ in time? Even for more sustained experiences causality is hard to prove as there are so many influences on a child attainment or attitudes. Measuring impacts on life outcomes is an enormously challenging task, yet many arts organisations are required to provide hard evidence of these in the battle for funding.
The National Literacy Trust has developed a framework to compare the outcomes of its range of projects for different audiences. This will be available for reference for other organisations in the sector.