Appendix C: evaluation matrix summary

The Evaluation Matrix below provides a summary of the key insights identified through the qualitative interviews and online survey, and are described under each major evaluation outcome theme.

Key Outcome: Organisational cultures are supportive of urban design

Key findings/evidence: Urban design promoted internally/externally  

Qualitative findings

  • Many participants agree that the Urban Design Protocol (Protocol)  is important in terms of improving communication and collaboration, and was being used by some organisations as an effective mechanism for educating/increasing awareness
  • The physical certificate can be seen as a tool for promoting urban design internally. One participant claimed the CEO Urban Design Champion (Champion) had the certificate placed on the wall in the cafeteria
  • The degree of promoting urban design varies across participants. One participant identified networking internally was an effective mechanism for increasing support within the organisation, while some identified frustration and burnout
  • Ensuring information filters through to all levels of the organisation is a challenge for some organisations
  • Many participants identified the Champion (as a Councillor) was the best persons for promoting urban design externally given their high profile

Online survey findings

  • Champions have been identified as promoting quality urban design within their organisation, with two out of three (67 percent) non champion respondents reporting this was the case
  • Changes in behaviour and/or thinking have occurred in many organisations as a result of the Protocol with two out of three respondents (67 percent) reporting this change. A further 12 percent reported that changes in behaviour and/or thinking were already occurring in their organisation before the Protocol was established
  • People within the organisation (94 percent) were identified as being a key driver of changes in behaviour and/or thinking regarding urban design, followed by the community (42 percent), other consultants (35 percent), professional associations (35 percent), and clients (35 percent).

Key findings/evidence: Support for urban design has increased

Qualitative findings

  • This research clearly reveals that the Protocol, and the use of case studies as best practice, has been effective in assisting organisations change behaviour and thinking
  • Some participants identified the changing roll of urban design, and now seen as more of a “team” approach
  • One organisation described their shift in thinking – to a greater picture, a more holistic approach to urban design. This shift also includes internal discussions
  • Some organisations have experienced a closer working relationship with other departments in their organisation
  • It was noted that support for urban design has extended to the wider market and barriers inhibiting good urban design practice were being removed

Online survey findings

  • Attitudes towards urban design are largely positive with high levels of agreement regarding taking a multi-disciplinary approach (79 percent), having strong support from senior management (79 percent), having quality urban design considered to be extremely important (76 percent), and a quality urban environment being a key priority in their organisation’s outcomes (75 percent)
  • Of note, just over half (58 percent) agreed  that their elected representatives strongly support urban design, and a further 18 percent were unsure or did not respond.

Key findings/evidence: Principles are applied

Qualitative findings

  • Some participants identified seeing the value received from the Protocol, and becoming part of their culture to apply the principles
  • It was identified that the seven C’s have gained a good reputation

Online survey findings

  • The urban design concepts and/or principles themselves appear to be having a strong influence on quality urban design. Three out of four respondents (77 percent) were in agreement that the urban design principles were being applied in their work/projects, and more than two out of three respondents (70 percent) agreed that urban design concepts were influencing new policy development

Key findings/evidence: Increased knowledge and awareness of good urban design

Qualitative findings

  • Increasing the knowledge and awareness of good urban design throughout the organisation is a challenge for some participants
  • Many participants identified that more education and training is needed
  • Mixed feedback was received regarding the suitability and quality of urban design tertiary courses, warranting further investigation
  • It was noted that urban design was being promoted through the Foundation for Research, Science and Technology (FoRST)
  • The Protocol has also created a greater understanding and interest in other areas, for example the building space
  • Some participants commented that while the Protocol has assisted in increasing awareness and understanding, it has had minimal impact on day-to-day business or how projects are conducted
  • Some organisations have also experienced an increasing interest of urban design in relation to the Resource Management Act (RMA) and other factors, indicating a change in behaviours

Online survey findings

  • Changes in behaviour and/or thinking have occurred in many organisations as a result of the Protocol with two out of three respondents (67 percent) reporting this change. A further 12 percent reported that changes in behaviour and/or thinking were already occurring in their organisation before the Protocol was established

Key findings/evidence: Influence strategic decision making

Qualitative findings

  • It was noted that the Protocol has been effective in influencing the teaching programme for urban design related courses
  • This research has revealed that while the Protocol has made significant changes regarding the awareness and understanding of urban design in general. It is still however, too early to identify whether the Protocol has been successful in influencing strategic decision making

Online survey findings

  • The Protocol’s influence in strategic decision making is having some traction, with slightly less than half  of all respondents (48 percent) agreeing that the Protocol was influencing the strategic decisions being made in their organisation, while one in three respondents (34 percent) reported this was not the case

Key findings/evidence: Investment into urban design capability/specific projects

Qualitative findings

  • A comment was made about the difficultly in linking the protocol to specific projects
  • Online survey findings
  • A total of $416 million was identified as having been directed towards urban design capability or specific projects to date. This equates to an average of $12.6 million per respondent organisation.

Key Outcome: Organisations demonstrate commitment to the programme

Key findings/evidence: Identifying signatories/Urban Design Champion

Qualitative findings

  • It was noted that the process of signing up to the Protocol (including identification of a Champion, and signatory action plan) was a straight forward process
  • This research revealed the process of appointing a Champion varied amongst organisations. Some Champions were selected given their high profile or status (e.g. Councillors, Chief Executive), while others were appointed given their role or “passion” for urban design (e.g. urban designer)
  • While Councillors appear to be an effective choice for a Champion (given their high profile), some participants expressed concerns regarding their long-term sustainability particularly given the nature of their term of office (i.e. three years)
  • Many organisations have also adopted different levels of Champions, one being a figure head and the other level being more of an internal representative or driver
  • More clarity is needed regarding the Champion role and expectations - many participants were unclear on what this means
    • The pool of urban design expertise is relatively small with some participants expressing concerns about the ability to find  a Champion with right qualities, both as a figure head or an internal representative
    • This research also identifies that some organisations have moved away from having one Champion to a “championing organisation”

Online survey findings

  • The process for selecting a Champion varies between organisations. The most frequently mentioned selection criterion includes their specific role and/or position within the organisation (46 percent), their level of seniority (45 percent), a passion/interest for urban design (42 percent), specialist urban design expertise (37 percent) and their high public profile/presence (23 percent)
  • The length of time Champions have held their role also varies. One in three respondents (31 percent) have held this Champion role for between one and two years, while a one in three (29 percent) have held this role for more than three years. Of note, one in six Champions (16 percent) reported being in this role for less than 12 months

Key findings/evidence: Completion of action plans

Qualitative findings

  • Nearly all organisations had completed their initial action plan
  • Resources used to complete the action plan included the Action Pack for action plans, and using other organisations’ action plans for case study
  • A comment was also made regarding the overall relevancy of action plans to small consultancies
    More guidance is needed for some organisations in relation to developing action plans, and the availability of models/examples

Online survey findings

  • The overall value of action plans have not been fully realised for about half of all signatory organisations. For example, half of all respondents (53 percent) were in agreement that the urban design action plans were an effective mechanism for demonstrating their commitment to the Protocol, while one in three (29 percent) did not agree that they were at all effective in this regard

Key findings/evidence: Experiencing value from undertaking action plans

Qualitative findings

  • Nearly all participants agree that action plans are a good tool for demonstrating commitment to the programme
  • One participant questioned the overall effectiveness of the action plan for private sector organisations
  • A concern was also raised in relation to the compliance aspect of undertaking action plans rather than adding value

Online survey findings

  • Half of all respondents (52 percent) agreed that undertaking their urban design action plans were of high value, while one in three (31 percent) did not agree

Key findings/evidence: Action plans have been achieved

Qualitative findings

  • Many organisations claim to have either completed their initial actions, or the actions were ongoing (i.e. fluid and/or evolving)
  • The key challenge for many organisations was updating the action plans with new actions
  • Staff resourcing was identified as a barrier to completing and implementing the action plans for some organisations
  • Comments were made regarding monitoring and evaluating action plans as a greater incentive for updating plans and implementing actions

Online survey findings

  • Main factors contributing to the success of an urban design action include:
    • education and awareness of urban design
    • having staff commitment
    • collaboration and involvement
    • a multi-disciplinary approach
    • a change in thinking
    • influencing policy development, or having clear policies or guidelines
    • producing high quality work in general.
  • Main barriers preventing the success of an urban design action included:
    • a lack of funding and resources
    • limited education and awareness
    • lack of collaboration or involvement
    • lack/limited ongoing commitment
    • lack/limited change in thinking
    • legislation/policy restrictions
    • maintaining strategic focus
    • low levels expertise/relevance.

Key findings/evidence: Projects/programmes improving urban environment have been implemented

Qualitative findings

  • No information collected


Online survey findings

  • See comments above.

Key findings/evidence: Other actions/programmes have been implemented

Qualitative findings

  • Comments were made regarding district plans not supporting good urban design, warranting further investigation.


Online survey findings

  • An average of 2.9 urban design work/programmes have been completed by each organisation that were not included in their action plan. While this indicates that many signatory action plans require updating, it also suggests that urban design work/programmes are being conducted regardless of whether or not they have been included in the urban design action plan

Key Outcome: Increased programme participation

Key findings/evidence: Attendance at symposiums and seminars

Qualitative findings

  • Most participants in the groups and interviews claim to have attended the symposiums and seminars
  • This research also revealed that given the current role/position of most signatories (i.e. Senior Management, Councillors), the opportunity to attend the symposiums is also available to wider group of employees (i.e. delegated)
  • Overall feedback received from the Urban Design Symposium is extremely positive-a good learning and networking environment, relevant topics and influential speakers
  • Some participants also commented that the topics/themes needed to be more targeted/relevant to the audience
  • A concern was also raised regarding the accessibility of speakers for questions or further discussion
    Location of symposiums can be a barrier regarding attendance (i.e. travel)
  • A comment was also expressed regarding the general lack of Urban Design Forum events in New Zealand, and the lack of networking opportunities for those outside the Auckland region
  • This research has also noted that only some Champions have organised (or plan to organise) their own seminars/forums, demonstrating they are being effective in their role regarding raising awareness and understanding of urban design

Online survey findings

  • Urban design related networking events and seminars are providing value to those who attend, with more than four out of five respondents (85 percent) rating the Protocol events they have attended as being of value.

Key findings/evidence: Use of Ministry’s Urban Design publications

Qualitative findings

  • Overall feedback received regarding the newsletter is also positive, and the tabloid styles articles were well received
  • Concerns were raised about the overall level of awareness of the Urban Leader e-newsletter, and potentially not reaching certain audiences within a signatory organisation
  • Overall feedback regarding other Ministry publications was also positive
  • Case Studies were considered by nearly all participants as relevant and useful. Having said this, many participants identified the need to have more relevant New Zealand Case Studies
  • Concerns were also raised about the need to include case studies or examples of bad urban design (i.e. what pitfalls to avoid, how to correct etc.), and a bottom-up/pedestrian approach
  • A comment was also expressed regarding using the web as a platform for case studies – less cost, greater depth, and greater accessibility

Online survey findings

  • More than three out of four respondents rate the following tools and resources of value:
    • Urban Design Toolkit (89 percent rate this extremely/somewhat valuable)
    • Urban Leader monthly e-newsletter (85 percent rate this extremely/somewhat valuable)
    • Urban design case studies (81 percent rate this extremely/somewhat valuable)
    • The Action Pack for action plans(79 percent rate this extremely/somewhat valuable)
    • The Review of Urban Design Case Law (77 percent rate this extremely/somewhat valuable).

Key findings/evidence: Urban design champions are effective

Qualitative findings

  • A comment was expressed regarding a case study of how different Champions operate in their organisation
  • Mixed feedback was received regarding the overall effectiveness of Champions. Some participants claimed their Champion was ineffective or ad-hoc with regards to increasing awareness among the community, while others claimed this was not the case. This is perhaps due to varied levels of knowledge and expertise of urban design, and a lack of clarity regarding expectations of their Champion position
  • Some participants identified that political Champions need to report back on progress in order to be more effective in their role
  • Some identified the need for more training for Champions
  • Concerns were raised about the issues of motivation, and conflicts of interest, in relation to the voluntary nature of the Champion’s role and their overall effectiveness

Online survey findings

  • Two out of three non-champion respondents (67 percent) reported their Champion promoted quality urban design within their organisation, while the remainder were predominately unsure (24 percent unsure, and 3 percent reported they did not promote quality urban design).
  • Champions provided a variety of examples of how they promoted urban design within their organisation. The main themes included:
    • providing quality, best practice advice and guidance
    • participating in discussion groups/forums
    • providing staff training and development
    • publishing articles, case studies, and research
    • general advocacy
    • involving clients in the process
    • sharing information
    • providing education and training
    • undertaking project reviews were
    • collaborating with others (internally and externally)
    • ensuring projects have an urban design strategy.

Key findings/evidence: Recipients network off-line

Qualitative findings

  • Many participants place a high value of networking, and exposure to other urban design professionals. This research also reveals that there is very limited networking outside of main organised events (viz. Symposiums, seminars, forums)
  • A comment was expressed about setting up the Urban Design Forum in regional centres as a special interest group

Online survey findings

  • Networking groups or activities not organised by the Ministry were considered to be of high value (a total of 90 percent rating this of value), as with other seminars and symposiums (a total of 92 percent rating this of value).

Key findings/evidence: Increased exchange of information and research

Qualitative findings

  • It was noted that Wellington did not have an urban design panel, compared with Auckland, Christchurch and Queenstown
  • The symposium environment was also considered to be an optimal platform for encouraging discussion and exchanging information

Online survey findings

  • Considerable value is also being obtained from exchanging urban design related information and research and (93 percent).

Key Outcome: The quality of urban design is enhanced

Key findings/evidence: Case studies of better quality urban design

Qualitative findings

  • It was noted that there was a shortage of good New Zealand case studies
  • Comments were also expressed regarding the usefulness of Australian case studies, while American case studies were considered to be irrelevant

Online survey findings

  • Urban Design Case Studies for Local Government were rated as being somewhat or extremely valuable by 81 percent of those who reported using this resource.

Key findings/evidence: Support of other organisations

Qualitative findings

  • It was noted that Greater Wellington has demonstrated a high level of support of other councils, and is seen as a leader in urban design
  • Influencing other organisations was identified as being difficult
  • Some organisations claimed to be working with other signatory organisations
  • An idea was expressed about co-sponsoring with other organisations to create joint local government urban design panels

Online survey findings

  • Considerable value is also being obtained from supporting other organisations (86 percent).
  • Examples of how signatories support other organisations varied, and included the following:
    • providing advice and guidance
    • that working in collaboration with others
    • participating in urban design related discussion groups and forums
    • sharing information in general
    • presenting at conferences or workshops
    • developing design guides
    • that providing education, training, and presentation

Key findings/evidence: Leveraging off Protocol to achieve projects

Qualitative findings

  • Some participants commented on the need for funding and publishing research

Online survey findings

  • Half of all respondents (53 percent) agreed that their organisation were using or leveraging off the Protocol to achieve their projects.

Key findings/evidence: Development of organisations own urban design framework/policies

Qualitative findings

  • It was noted that Christchurch were developing an urban design development strategy
  • A comment was received about the benefits of each town and village having their own urban design strategy

Online survey findings

  • More than half of all respondents (58 percent) agreed that having their own urban design guidance was of high value. In line with this, one in two respondents (48 percent) reported their organisation had either developed or were in the process of developing their own urban design guidance. Of those who were unsure or had not yet developed any guidance (36 percent), two out of five reported their organisation was likely or extremely likely to start developing this over the next 12 months.

Key Outcome: The Ministry positively contributes to the success of the programme

Key findings/evidence: The Ministry’s tools and resources are effective

Qualitative findings

  • This research reveals that participants consider the Ministry’s tools and resources to be effective. This is also evident in earlier comments regarding case studies and newsletters
  • Some participants also commented on the Ministry’s success despite the lack of funding/under resourcing
  • It was noted that some of the Ministry’s communications were not reaching the community level

Online survey findings

  • The Ministry’s more tangible tools and resources (i.e. published documents etc.) are rated of greater value than those of a less tangible nature (i.e. services). The Urban Design Toolkit, Urban Leader e-newsletter, Urban Design Case Studies, the Action  Pack for action plans, and the Review of Urban Design Case Law were rated as being extremely or somewhat valuable by more than three out of four respondents (between 77 and 89 percent)

Key findings/evidence: Action Pack and guidance is effective

Qualitative findings

  • Of those participants who were initially involved in the development of their organisation’s action plan, identified the Action Pack and Ministry’s guidance was effective
  • Only one participant claimed they did not use the Action Pack

Online survey findings

  • See comments above.

Key findings/evidence: MfE provides effective support in the development of action plans

Qualitative findings

  • Most participants agree that the Ministry has contributed significantly towards increasing awareness and knowledge about urban design

Online survey findings

  • The support and guidance received from the Ministry in relation to the development of action plans was rated being of value to three out of five respondents (61 percent), while one in five (22 percent) did not find this of value

Other findings of interest/future suggestions

Qualitative findings

  • It was also noted that the Ministry has been effective in influencing urban design professional development and education, such that New Zealand no longer needs to import urban design specialists
  • Need to continue raising awareness of urban design in general, and improve understanding of how to handle density
  • Raising awareness of urban design among the general public through a television programme
  • Including a personal assessment in the case studies Creating streams in terms of Champions and their expertise
  • The Ministry is best placed in assisting organisations become more co-ordinated, and educating the public with regards to the benefits of good urban design
  • Providing greater guidance to smaller towns/rural areas
  • Funding smaller councils in order to attend the five day workshops
  • Introducing a CABE model, or UK style of planning
  • Creating a national urban design panel or joining local government urban design panels
  • Having a greater commitment from central government
  • Ensure more research is conducted or made available
  • Providing more training and/or mentors for Champions

Online survey findings

  • Continues to promote the Protocol and the benefits of quality urban design in order to further enhance the changes in behaviour and thinking that have occurred to date
  • Continues to provide Protocol events, and consider increasing the number of events being provided in regional locations
  • Continues to further develop the urban design tools and resources (viz. Urban Design Toolkit, UrbanLeader e-newsletter, Urban Design Case Studies, the Action Pack, the Review of Urban Design Case Law)
  • Promotes the value organisations receive through exchanging urban design related information and research, and the value received through networking with other organisations
  • Promotes the benefits and value (including leverage) signatory organisations receive from achieving their action plans
  • Develops a strategy for working closer with the central government sector, building networks, promoting the Protocol and raising awareness of quality urban design
  • Develops a strategy, in conjunction with education providers, to address urban design capacity issues and consider seeking input from the wider industry
  • Develops a strategy to assist signatories develop their own urban design guidance, and consider the development of tools and resources to aid signatories in this process
  • Reviews the Champion resources in order to provide a more tailored support for existing and new Champions, and senior management
  • Reviews the action plan resources in order to provide more tailored support and advice for signatories developing, and maintaining, their action plans
  • Reviews the current administration of signatory action plans (including its database of signatory organisations) in order to make this a more efficient and effective process, and allows for greater communication between signatories and the Ministry in regards to ongoing monitoring of action plans
    Targets key influential or strategic networks or stakeholders to sign up to the Protocol
  • Considers optimising the Urban Leader e-newsletter as a tool of communicating with signatories (including online linkages), and facilitating greater two-way communication
  • Considers updating the Protocol (including visual graphics) to ensure it remains current and relevant
  • Finally, keep up the good work and target the areas identified as needing attention.