2 Methodology
2.1 Description of Methodology
Survey Design
The survey material consisted of interview questions and discussion prompts that were specifically developed with reference to the competencies, learning objectives and elements of good (and bad) practice identified by the MGDP. The final survey material was agreed to and approved by the Ministry for the Environment to ensure the essence of the study purpose was appropriately reflected.
A copy of the survey material is attached as Appendix 1 to this report.
The survey consisted of six key questions relating to key aspects of the hearings and decision-making process. For each question, respondents were asked to comment on any key areas of change observed during hearings. Each question focused on drawing out three key types of information as part of the interview:
- Discussion prompts enabled the interviewer to categorise the type of change observed in relation to the key competency areas of the MGDP. This provided a level of quantitative guidance on the proportions of hearings commissioners and hearings committees exhibiting such changes.
- Respondents assigned 'change ratings' for each change observed. This provided average ratings with regard to:
- The extent of the change observed. Here, respondents were asked to comment on whether the degree of the change observed was perceived as minor, moderate or major. This provided the interviewer with a quantitative guide to the degree of change in performance exhibited.
- The importance/significance of the change observed. Here, respondents were asked to comment on whether the change observed was perceived to be of minor significance, moderate significance, or major significance. This provided the interviewer with a quantitative guide as to the perceived importance of the change observed in relation to the learning competencies and aims of the MGDP.
- Qualitative comments were also recorded to illustrate key points and key themes that emerged during the interview process.
Selection Process
A list of 40 survey respondents was developed to represent a reasonable cross-section of resource management practitioners with respect to their roles, experience, practice environment, and to represent a reasonable number of hearings commissioners and hearings committees across the nation to which their observations would relate.
The selection process involved four key steps:
- Preliminary contact was made with a senior staff member at a representative range of local authorities around New Zealand to identify an appropriate consents manager/reporting officer within council who would be qualified to participate in the survey.
- The consents manager/reporting officer was asked to provide referrals of names of consultant planners, resource management lawyers and surveyors who regularly attend resource consent hearings in their district on behalf of clients.
- A list was prepared with the names of all potential respondents. A total of 40 were selected to ensure a good geographic spread. The list of 40 potential respondents was provided to the Ministry for agreement and to ensure there were no potential conflicts of interest (such as participants being on relevant advisory boards).
- Contact was made with each of the 40 respondents and a copy of the survey material was provided in advance of the interview to allow adequate preparation time and to maximise the nature of responses. In the event that a respondent was unavailable or unwilling to take part, an alternative was selected from the initial master list.
It was important that each respondent met the following three key selection criteria:
- The respondent must be an experienced practitioner in the resource management field.
- The respondent must have at least three years experience before local resource consent hearings, and therefore regularly observe the performance of decision-makers (either on behalf of clients or as a consents manager and/or reporting officer within a local authority) in relation to notified resource consent applications.
- The respondent must have regularly attended notified resource consent hearings during 2004, 2005 and 2006.
Interview Process
Generally, the surveys were undertaken via a short telephone interview. However, a few respondents preferred to provide written comments. All responses were recorded and collated within an Excel spreadsheet database.
2.2 Potential Limitations with the Methodology
There are some areas in the described method to be mindful of when interpreting results presented in this report.
- There was a general apprehension across respondents to provide specific comments on the performance of individual committee members or commissioners. Often, responses were generalised to the committee membership as a whole. In some cases where respondents were providing observations on more than one hearings committee, comments were provided on a collective basis across all panels observed. In any case, it is noted that respondents' comments are quoted anonymously throughout this report, and all respondents were advised of this prior to taking part in the survey.
- There was some difficulty identifying the most appropriate reporting officer within some of the larger city councils approached to participate. In these councils it is not common for a planner to have gained the continuity of observation of one particular hearings panel given the volume of applications that go to hearing and the various combinations of hearings panels.
- To some extent, the final list of survey respondents evolved as the study progressed and was dependent on the availability of respondents. Generally, local authority staff were more available to take part than consultant planners or resource management lawyers.
- Many respondents reported difficulties using the nominated rating scales. For example, many indicated that it did not work well for those hearings commissioners and committees who were operating at a high level prior to training. A 'before and after' rating scale may have worked well in these instances.
