Measuring the success of your product stewardship scheme
March 2011, Ref. INFO 585
This information sheet is for anyone developing or operating a product stewardship scheme. It explains why evaluating your scheme's success is important and gives some tips for setting objectives and targets. An example of measuring a product stewardship scheme is also included.
Why evaluating your product stewardship scheme is important
Evaluating your product stewardship scheme is an important part of the product stewardship accreditation process. Evaluation starts at the planning stage of your scheme before you apply for accreditation. To qualify for accreditation under the Waste Minimisation Act 2008 your product stewardship scheme must:
- set measurable waste minimisation, treatment, or disposal objectives for the product
- set timeframes for meeting the objectives.
You must also be able to show that you have a process for assessing the scheme’s performance and reporting on its performance to the Minister and the Ministry for the Environment.
Evaluation is important to measure progress, check if your product stewardship scheme is running effectively, and tell others of your successes. By evaluating your scheme you will be able to:
- communicate a shared understanding of what you intend to achieve
- produce tangible proof of how well your scheme is working (as intended or not)
- identify learning’s and areas needing improvement
- assess and report on your scheme’s performance.
Tips for setting objectives and targets for your product stewardship scheme
To measure the success of your scheme you will need to set objectives and targets. Below are five tips to help you do this.
1. Decide what you want your scheme to achieve
What are your most important goals and where do you want your scheme to head in the future?
2. Evaluation planning
When setting up your product stewardship scheme, identify what information you will need to measure your progress and success. For example:
- activity measures (efficiency) – delivery to timeframes/within budget, customer or staff feedback on quality, levels of satisfaction or number of complaints or non-compliance
- immediate impact measures – number of participants before and after, levels of support, understanding of your product stewardship scheme by customers, participants and staff
- longer term impact measures – tonnage of waste before compared to tonnage after, number of your products recycled, monetary contribution to the local economy and number of new jobs created, environmental benefits from the scheme, such as improvements in air or water quality.
The information needs to be clear, based on facts not vague claims, scalable to the size of your project, relevant, available and usable.
3. Baseline data
It is important that you capture a baseline measure to compare progress against. You will need to find data that is relevant, timely (ideally less than 5 years old), and that can be updated annually for you to report on. Examples of information that might be useful include:
- qualitative data – for example, asking for information in a survey, interviews, or focus groups. This information can be useful to evaluate before and after changes in levels of awareness, number of scheme participants, and customer support, satisfaction and behaviour change
- quantitative data – ‘hard’ numbers – for example, forecast proportions, market coverage, waste volumes, compositions, cost/profits. Trends over the life of the scheme such as an increase or decrease of your product in a year, increase of foot traffic in store, increase of media coverage of your product stewardship scheme, frequency of collections, number of collection sites, and sales of certain products.
This information can be useful to evaluate efficiency, immediate impacts, and end/longer term impacts.
4. SMART criteria for ‘good’ objectives
Objectives are:
- concise statements that answer the “so what”?
- measurable and implicitly have a specific scope and timeframe
- the key benefit and end result you expect the scheme to achieve and how you’ll achieve it.
Below are some examples of bad and good SMART objectives.
Specific
x Customers will be aware of waste minimisation
ü75% of customers reported high awareness and support for waste minimisation after purchasing our product and speaking with our staff in store compared to non-customers
Measurable
x More <product> recycled due to the product stewardship scheme
üNumber of take-back vouchers used by retailers to return <product> for recycling increases from 5% [benchmark] to 65% in the next year
Achievable
x Increase recycling from 5% to 100% within a year
üAmount of <product> recycled increases from 3 to 10+ tonnes per year within two years
Realistic/relevant
x Feasibility study will identify ways to reduce <product waste> to landfill by 50% within two years of operation
üFeasibility study identifies over 90% of potential <product waste> sources in North Island that could use the service
Timely
x <Future technology> will manage 90% of sorting by 2040
ü<Technology> incrementally increases the proportion of waste sorting by 5–10% annually for the first 2–3 years
5. Tips for measuring reduction in waste
- Reduction can occur as recycling, reuse, recovery, or avoidance. Identify which one or combination of these you need to measure.
- Ensure you have baseline information so you have something to compare your scheme's progress against.
- Be specific, so you know what you are measuring and record this data at regular time intervals.
Other sources that may help:
- The Ministry for the Environment’s ‘A Guide to Product Stewardship’
- Statistics NZ, local councils, WasteMINZ, the Sustainable Business Network, industry associations and waste consultants such as SKM or WasteNotConsulting may have conducted surveys or collected waste-related or industry information relevant to your scheme.
- www.surveymonkey.com is a free (or cheap) website through which you can design and roll out an online survey. It includes common questions and survey templates.
- WRAP (the Waste and Resources Action Programme) is a major UK programme and has free web resources on monitoring and evaluating waste diversion schemes.
Example of measuring a product stewardship scheme
This example demonstrates the types of measures and associated objectives and targets that your product stewardship scheme could evaluate. It is also a good summary of what to include in your annual progress report to the Minister and the Ministry for the Environment.
Measure
(what are you measuring?)Specific objective/target
(for example, percentage, tonnage, number of products collected, including projected future targets)
Note that these can change from year to year and progressive, ambitious targets over the life of the scheme are desirable.Timeframe
(when will this target be achieved?)Achievements in [insert year]
Report annually on your achievements to date to the Minister and the Ministry for the Environment (required); and to your participants/stakeholders (desired). It is important you provide information on ALL of your targets, even if you have not met them.<Product> recycled/recovered/reused due to product stewardship scheme
- Number/volume or percentage of total of <product> that is recycled/reused
- Percentage or number of <product> returned for reuse
- Quantities of materials requiring disposal (eg, not suitable for reuse/recycling)
- Number of items assessed as reusable
Weekly, monthly, annually
Contribution to community
- Monetary contribution for recycling initiatives (eg, $5000 per annum)
- <Product> contribution (eg, 10,000 recycled <product> to <specify community> or nationwide
- Number of new jobs (eg, 20 new employment opportunities in <specify community> or nationwide
- Partnership initiative with local business and/or iwi to improve local outcomes (social, economic, cultural, environmental)
Weekly, monthly, annually or specify date
- How much to date (eg, 30 March 2011) or how much weekly, monthly, annually etc
- How many to date (eg, 30 March 2011) or how many weekly, monthly, annually etc
Customer awareness/support of scheme or measuring successes of customer awareness programme
Consumer and staff understanding of scheme
(Examples of how this information may be gained are a consumer survey, feedback forms and focus groups)
- Percentage or number of specific town/city aware of scheme (general public)
- Percentage of New Zealanders aware (general public)
- Number of individuals/organisations who seek out/use the scheme
- Amount of media coverage
- Number of visits/talks given and number of people spoken to at these
- Number of vouchers distributed compared to the number of vouchers redeemed
- Number of new first time users of the scheme
- Consumers level of understanding (eg, 50% of consumers fully grasp scheme)
- Percentage of staff understanding at all levels of the organisation
Annual review
- Report on the specific target
Support from or engagement/involvement with a specified sector eg, local government
- Percentage or number of <sector> eg, <councils> represented and ratepayers reached
Annually
- How many to date eg, 35 councils involved
Customer satisfaction
- Percentage of positive feedback received from customers
Reported on weekly, monthly, annually etc
- Survey, customer feedback form results or verbal feedback recorded
Coverage/completeness of the scheme
- Percentage of industry participation by number and/or sales/volume
- Percentage of geographical area covered by scheme
- Number of participants
For example, by March 2011
Environmental impacts/benefits from the scheme
- For example
- air emissions – reduction in dioxins released, nitrogen oxide content reduced
- water quality – heavy metal contaminants reduced
- soil quality – reduction in heavy metal levels
- Scheme participants’ compliance with Resource Management Act consents
Show annual or bi-annual improvements
- Document measure improvements since scheme begun
Last updated: 25 March 2011
