(Impact) evaluation and assessment framework

From Urban Arena Wiki
Jump to navigation Jump to search

Research on sustainable and just urban areas has involved a variety of evaluation and assessment methods. This cluster summarizes a sample of these methods, which have been employed in the study of the following topics: environmental conflict in coastal urban areas, food supply chains, transitions to sustainable and low-carbon societies, environmental public health risks, distribution of green amenities, and common good contributions by companies and other organizations. What unites these approaches is not their substantive content, but rather the innovative, fruitful, and transferable methods that can be employed in many contexts.

This page is part of an ongoing, open-ended online collaborative database, which collects relevant approaches that can be used by city-makers to tackle unsustainability and injustice in cities. It is based mainly on knowledge generated in EU-funded projects and touches on fast changing fields. As such, this page makes no claims of authoritative completeness and welcomes your suggestions.

General introduction to approach

While each evaluation/assessment method discussed here is highly unique, they share several commonalities. Each one includes in-depth research on their issue and its context. This could include research on governance arrangements and the impacts of certain events, system change, development patterns etc. The assessments are carried out using case studies, pilot projects, and in one case, big-data. Each assessment framework is mainly aimed at helping policymakers by: producing policy recommendations, decision-support tools, public-use data repositories, scenarios, enhancing policy learning, predicting and mitigating risks, and even changing norms in policy making. Each assessment method has a strong focus on environmental sustainability, and a varying degree of justice considerations.

Note: Two approaches, Socio-technical strategy assessment (DESAFIO, 2013-2015)[1] and Impacts quantification of globalization (GLOBAL-IQ, 2011-2014)[2], are worth noting but do not sufficiently cover UrbanA concepts of sustainability and justice in urban settings to be further elaborated upon here. See references for more information about the projects.

Shapes, sizes and applications

Environmental Conflict Assessment Framework (CAF), SECOA project (2009-2013): [3])

The CAF was developed and tested through the SECOA project on 17 coastal areas in Europe and Asia. It starts with elaborating on resources and resource users, including influence and power relations between users. CAF then identifies specific conflicts that are triggered by competing resource use, and typifies, ranks, and analyses all stakeholders involved in the conflicts. This assessment contributes towards conflict mediation and resolution, along with building scenarios and producing a policy recommendation toolbox. Conflict assessment frameworks, in general, are a well-established tool, and the breadth of SECOA’s test areas and conflict types indicates that the approach is transferable to many coastal urban contexts. However, as a broad, region-based framework tool, it must be contextualized for each urban area in order to be useful.

Food-Chain Analysis with a focus on SMEs, FOODMETERS project (2012-2015):[4]

Food-chain analysis involves the use of various assessment tools targeted at understanding urban food systems, with a focus on small and medium enterprises. Analysis includes the Metropolitan Footprint Tool, developed by FOODMETERS to measure environmental impact of urban food consumption and estimate self-sufficiency. Analysis also includes assessing the impact of different food chain types, e.g. short food supply chains, on sustainability and resilience of urban regions. Regarding the maturity, successes, limitations and transferability of this approach, please see the Sustainable Food Supply Chains cluster.

Initiative-based learning (IBL), PATHWAYS project (2013-2016): [5]

IBL is a qualitative assessment tool that looks at the evolution, governance (including actor constellations in various institutional settings), and scaling of sustainability transitions through the lens of specific cases (initiatives). It considers technological, economic, social and ecological dimensions. IBL provides a framework to understand how specific initiatives influence broader contexts, as well as reveal the complexity and uncertainty of sustainability transitions. IBL was successfully used in PATHWAYS to provide an analytical framework of sustainability transitions in specific cases within different sectors.

Predictive system of public health focused on risk and resilience, PULSE project (2016-Oct 2019): [6]

This approach uses geolocated, population-based data, including data gathered by citizens through smart technologies, data from city governments, and from health systems to undertake spatio-temporal health impact assessments of environmental risks. This includes identifying health inequities across cities. The assessment aims to predict and intervene against health risks such as asthma and type two diabetes related to issues like air quality and mobility within cities. The PULSE project is championing this approach, so it is relatively new and its successes and limitations are unknown.

Fair Urban Greening (FUG) Index, GREENLULUS project (2016-2021): [7]

The Fair Urban Greening index measures the (in)equity of distribution of green amenities resulting from the greening of locally unwanted land uses both in and across cities, in order to assess the social and racial impact of new green amenities in distressed neighborhoods (e.g. gentrification) to counterbalance pure environmental assessments. This approach is novel, since, according to the GREENLULUS project, no large scale study of the relationship between green cities and racial/social equity has been conducted. GREENLULUS is currently applying the FUG index to 40 cities in North America and Europe.

Common Good Matrix, The Economy for the Common Good (n.d.): [8]

The Common Good Matrix, a value-oriented benchmark system developed for the institutional sector (companies, municipalities or other organisations) allows the assessment of their impact on the common good. This assessment tool positions increasing the common good (e.g. quality of life, social justice, environmental sustainability) as a main goal of organizations, rather than purely financial gain. This Matrix has been developed by The Economy for the Common Good, which has been endorsed by the European Economic and Social Committee. While the Matrix appears to be supported by important actors, a foreseeable limitation is that its success hinges on uptake among organizations which are not required to use it.

Further examples of how (Impact) evaluation and assessment frameworks can be used to create sustainable and just cities include: an in-progress framework (included in this paper which is being built to evaluate transformative capacity according to diversity and inclusivity parameters among others, the UnaLab project which is developing a common evaluation framework for NBS (based on environmental, social, economic factors) within the Nature4Cities project, and the City of Augsburg's ongoing development of sustainability indicators that are to be mainstreamed into its various City departments.

Relation to UrbanA themes: Cities, sustainability, and justice

Urban. Most of the assessment methodologies (including CAF, food-chain analysis, predictive system of public health, and the FUG index) are inherently linked to urban settings and issues, and the others (including initiative based learning and the common good matrix) could also be applied to non-urban areas. Europe, North America and parts of Asia are the studied urban regions.

Sustainability. All assessments are concerned highly with environmental sustainability, primarily at the urban or regional scale.

Justice. Many of the approaches explicitly address justice and some have less direct, but still present, justice implications. Direct considerations of justice in assessments include investigating power inequalities between resource users (CAF), health inequities across cities (predictive system of public health), distribution of green amenities and unwanted land uses among racialized communities (FUG index), and social justice impacts of organizations (common good matrix). Food-chain analysis is mainly focused on environmental impact, rather than justice, and IBL has a social, but no explicit justice focus. Both procedural and distributive justice are achieved through the use of participatory research methods and considerations of various inequities in urban regions.

Linking sustainability and justice. Half of the approaches (FUG index, predictive system of public health, and common good matrix) directly connect sustainability and justice, while the link between sustainability and justice in the others depends highly on their application in context.

Narrative of change

Pressing and novel challenges in cities related to environmental and justice concerns require well-informed and innovative policy responses. Evaluation and assessment frameworks aim to help inform policymakers and other decision makers and provide functional tools to address these concerns. These frameworks, which often produce data for the public, can also empower citizens to advocate for themselves and the environment based on awareness of the issues.

Transformative potential

In general, the transformative potential of evaluations and assessments is dependent on their perceived importance by decision-makers and/or their citizens. These approaches are tools to reveal problems and suggest ways forward, but they do not have inherent transformative potential. Provided that they make it to the agendas of urban areas, most approaches in this overview have transformative potential. For example, Food-chain analysis may challenge the power relations between large and small, local producers, IBL reveals how small experiments disturb the status quo and spark sustainability transitions, predictive health systems and the FUG index problematize race or place based discrimination in cities.

Participants of arena#1 in Rotterdam identified the following important aspects and opportunities associated with (Impact) evaluation and assessment frameworks: their ability to inform policy and empower citizens with open data, their ability to create co-learning experiences between different actors, and the importance of having shared and consistent evaluation frameworks across cities/regions.

The following concerning aspects and challenges of the approach were also discussed: ensuring that the data is as neutral as possible, difficulty measuring qualitative processes with quantitative data methods, limited comparability of findings across different contexts, the development of good and meaningful indicators and clear objectives of the framework, and the possible exclusion of marginalized voices while developing an assessment framework.

Illustration of approach

Initiative Based Learning: Case study of Brixton community energy project in London (From PATHWAYS project) [9] IBL was used to study the evolution of a cooperatively owned solar energy project in London, England, and the UK’s first inner-city renewable energy co-operative. PATHWAYS studied the gestation, development and implementation of the initial program, and analyzed its potential for replication and transfer across contexts and scales. After the smaller-scale project in Brixton proved successful, it became a not for profit [10] which is currently working to help other communities develop renewable co-ops. The IBL assessment included great detail on specific actor networks, financing, policy influence, community involvement and so on. Initial replication efforts were mainly peer-to-peer on a smaller scale, while later on the initiative was transferred to a larger scale by pulling information from its smaller projects and feeding it ‘upstream’ into the Repowering Ltd. organization. This case study is a good example of an initiative that connects sustainability and justice in an urban setting, and demonstrates how assessment methods like IBL are used to better understand the role of local initiatives in energy transitions.