Editor’s note:
This blog is focused on sharing stories and information about living with fire. Occasionally, we also publish content related to social-change networks: the “how” in our approach to fire adaptation.
Any system-shifting endeavor should consider the role of evaluation in their work. Complete with moving targets, changing conditions, long time horizons between now and the change we seek, and reliance on the actions of linked, but autonomous people, this work is hard to measure. It requires us to check course often as we steer toward a collective “north star.” Taking the long view—and working in a space where one-size-fits-all answers do not exist—means we’re more often looking for sign posts and indications we’re on the right track rather than measuring immediate outcomes.
This post outlines the approach FAC Net has taken to evaluating our impact and describes some preliminary analysis and results.
The Fire Adaptation North Star
We need a new approach to fire management. One that invests in place-based solutions to build community resilience, reduce fire risk, expand who uses fire to restore landscapes and address the systemic barriers to this decentralized approach. A set of learning networks, composed of partnerships among local leaders across the US, stewarded by The Nature Conservancy and the Watershed Center and supported by the US Forest Service and Departments of Interior, is working to shift fire management toward this new paradigm. These partners work together to create change on multiple levels. Our approach is an example of a multiscalar network described by June Holley as, “networks that cross levels or layers turning innovation into widespread systemic transformation.”
Designing the Evaluation
Instead of allowing networks to evolve without direction, successful individuals, groups and organizations have found that it pays to actively manage your network. – Krebs and Holley
One of the keys to “actively managing” a network is evaluation. FAC Net has been collecting member feedback since launching in 2013. We frequently make adjustments to our operations based on the needs articulated by members, but we realized that a more robust evaluation effort would benefit our work. In 2016, we began a multi-part evaluation process.
Evaluation methodologies run a wide gamut, from multi-year longitudinal studies with control groups seeking to describe impact, to individual survey responses. Because FAC Net is both a result itself AND a process to get to fire adaptation results , we had to consider how successful FAC Net is at being a network, and also the fire adaptation results it is enabling. We concluded that a three-part evaluation process following the “three pillar” model as depicted in the graphic above would yield information that could improve network management, as well as help us communicate the impacts of the network with funders and other stakeholders. Using this framework, we set out to:
- Measure our connectivity using Social Network Analysis (SNA).
- Gather anonymized quantitative data on FAC Net’s health using a custom scorecard.
- Describe some of FAC Net’s intermediate results and impacts with Ripple Effects Mapping (REM) and case study processes.
Connectivity: Social Network Analysis
Evaluating network connectivity answers questions like, “who is connected?” and, “how are connections changing over time?” In a social change network, like FAC Net, the goal is to create a strong core (lots of people with strong bonds) with a healthy periphery (lots of additional ties and loose bonds). Connections among members should strengthen over time, and newer or more loosely connected members who bring new ideas and potential, should be actively cultivated and “woven” into the system.
We adjusted the SNA survey in its second iteration. Where in 2016 we measured “potential for collaboration,” in 2018 we determined to instead measure “level of influence.”
Returning to the concept of multiscalar networks, mentioned above, June Holley describes three levels:
- Level 1: Build local networks for experimentation
- Level 2: Build networks for scaling out so that local innovations can spread, inspire, and learn from others
- Level 3: Build networks for scaling up so infrastructure and policy to support innovations can be developed
Starting the process by measuring “collaboration” obscured our ability to see the impacts FAC Net was having on the individual practitioners and their local work (Levels 1 and 2). Instead, we determined that measuring who was influencing each other would reveal how people’s work was changing as a result of being part of FAC Net, giving us insight into who was testing new ideas and experimenting, and how fire adaptation practices were “scaling out.” Our vision is for network members’ joint efforts to also catalyze change at the system level (Level 3)—bringing to light needed policy changes and shifting national conversations about fire management. But for the purposes of this evaluation, we wanted to better understand FAC Net’s impact on Level 1 and 2 work.
What did we learn from the SNA?
Close relationships often result in high levels of influence. Analysis showed that depth of relationship was closely correlated to level of influence, which is an important foundation in FAC Net’s theory of change. FAC Net posits that by connecting FAC practitioners and helping to facilitate relationships and trust among them, they will influence each other’s local work. The 2018 SNA shows a 0.81 correlation coefficient between connection and influence, where 93% of close relationships also exhibit moderate or major influence and 76% of all relationships also exhibit moderate or major influence. By investing in members and their relationships, FAC Net can influence local work and shape fire adaptation practices around the country.
FAC Net exposes practitioners to new contacts and strengthens their connections. When looking at the data from respondents that completed both SNAs, there was a substantial increase in the depth of members’ connections: 44% of all connections that weren’t already close relationships were deepened between 2016 and 2018. Additionally, 72% of all possible connections in the Network are considered “relationships” in 2018, compared to 39% of possible connections in 2016.
There is potential for growth—of the network, and of the relationships among members. When you include people who didn’t take the survey in 2016 (those that are newer to the Network), 2018 SNA data shows that 50% of all possible connections across the network are “relationships.” There is still plenty of potential for relationship-building within FAC Net.
SNA provides useful data to netweavers. FAC Net staff have used the SNA data as a netweaving tool. Better understanding who is connected, and who is not, has given us insight into where we have the potential to help catalyze new and powerful connections. More informed netweaving has resulted in both increased connectivity and the strategic transfer of practices, tools and ideas.
Health: Anonymous Scorecard
After reviewing a number of other network’s “health scorecards”, FAC Net staff created a custom tool to gather anonymous information from members about their perceptions of network health and function. Questions covered a range of issues including those focused on:
- Purpose and value of network (what is FAC Net’s purpose, what could increase your satisfaction with FAC Net, who else should be involved, what would you like to see FAC Net accomplish);
- Individual participation (quality of relationships, status of time, resources and permission needed to participate, use of network systems)
- Local work (increased vision of what constitutes fire adaptation, learning about and implementing new practices, improving local strategies, leveraging funding, increasing the profile of members’ work)
- FAC Net’s effectiveness (progress toward our purpose, network growth, decision-making, accountability, space for multiple perspectives, adequacy of network resources)
- Perceptions of one another (doing more together than we could alone, adding value to each other’s work, accounting for shared interest when acting, honoring commitments)
FAC Net has made it a practice to collect feedback from members at frequent intervals and use that feedback to adjust our operations. Evaluations of workshops, bi-annual reports and one-on-one conversations have provided critical insights into how to best serve members. However, the majority of this data has not been collected anonymously. To make space for more critical feedback, we conducted an anonymous heath scorecard survey. The scorecard was completed by 78% of network members.
What did we learn from the scorecard?
FAC Net members have valuable insight. As always, FAC Net members provided key insights into how to meet their needs, the aspects of the network that are providing the most value to them and priorities we could work on together in the coming years. FAC Net staff have used this information to inform operations, work plans and guide network strategy.
Relationships among network members are strong, and they influence work done on the ground. Questions related to members’ relationships with one another and network staff were overwhelmingly positive with 92% of respondents either agreeing or strongly agreeing that they have formed meaningful new relationships or strengthened existing relationships as a result of the network. Additionally, as a result of relationships with FAC Net members, 96% of respondents felt that they were more effective in their work.
FAC Net provides benefits to members and their organizations. Members are receiving personal and professional benefits from their participation. For example, 92% of respondents agree or strongly agree that they have learned new skills through FAC Net. All but one respondent reported that FAC Net participation has benefited their organization, “Being affiliated with the Network has helped our credibility and ‘opened a lot of doors’ in trying new ideas/strategies with my fire partners and stakeholders in my community.”
Fire adaptation work done in communities is influenced by FAC Net participation. Members are sharing ideas and initiatives related to smoke outreach and public health as well as long-term recovery and prescribed fire use, and these concepts are being spread through FAC Net. “I’ve gained efficiencies by borrowing and adapting materials…and find regular new scientific literature that increases my knowledge base for effective practice.” Eighty-four percent of scorecard respondents reported that they have implemented a new practice in their communities because of the network.
FAC Net resources, and those it helps leverage, are making more possible. In addition to member support and inspiration, funding to try creative or new strategies is an important part of FAC Net’s approach, “[FAC Net] funding has been critical to facilitate collaboration opportunities that our typical fuel’s grant wouldn’t allow. It helps us also go bigger with local coalitions.” Eighty-eight percent of respondents agreed or strongly agreed that FAC Net had helped them leverage funding they otherwise would not have been able to pursue.
Members achieve more together than they could alone. Ninety-two percent of respondents reported feeling that members were achieving more together than they could alone and 100% of respondents agreed or strongly agreed that members add value to each other’s work.
Results: Ripple Effects Mapping
Finding a methodology that could measure our impacts and results was the most challenging part of this evaluation process. The complexity of the outcomes we’re seeking, along with a number of other confounding factors, led us to determine that gathering anecdotal information about the impacts FAC Net was having from a sub-set of members was the most feasible approach.
Members of FAC Net from the Pacific Northwest during the Ripple Effects Mapping (REM) session
Working with an evaluation team from Washington State University, we conducted a Ripple Effects Mapping (REM) session with members of FAC Net from the Pacific Northwest. We knew these members had strong relationships and that they had influenced one another’s work following a series of in-person learning exchanges supported by FAC Net, but we wanted to document the specific details of these impacts. REM allowed us to do that.
The REM data, collected as a series of “mind maps” where participants built off “ripples” that elaborated on the outcomes resulting from a central activity, demonstrated how FAC Net is working in a multiscalar way. You’ll recall from the model shared earlier, that in multiscalar networks, “level 1” is focused on supporting “local experimentation.” Several of the mind maps created in the REM process illustrated this “chain of impact.” For example, in Southern Oregon, members recognized how wildfire smoke was affecting their community and economy. With FAC Net support, they launched “Smokewise,” a partnership between network members and their local health providers and chamber of commerce to address smoke preparedness and adaptation. This idea then spread. Members’ work on smoke issues in Washington State, Santa Fe and Northern California was informed by, and also influenced, the work Southern Oregon was leading. Smoke work was “scaling out” through FAC Net. Taking it one step further, Southern Oregon members describe how this initiative began “scaling up” to influence the system (level 3), “…we were invited to the governor’s office and we were able to share how to prepare for a smoke season. The governor came back to us {and shared that} ‘Smokewise’ may become a statewide structure.” This example illustrates how FAC Net supports multiscalar change: propelling members’ approaches from local experiment, spreading the ideas throughout the network, and ultimately garnering the attention of the governor’s office.
What did we learn from Ripple Effects Mapping?
REM confirmed that a central value of FAC Net participation is connectivity. Data collected about connectivity illustrates two distinct values members derive from peer connections. First, the moral support and sense of community it creates, “I would never want to do this work without {FAC Net} support and I probably wouldn’t if the network went away.” And secondly, access to advice from people who have relevant experience to share. “{FAC Net} provides you the opportunity to shoot someone an email, “is this {new approach} something I should invest my time in?”
Members’ priorities and local focus areas are changing because of the network. Members describe recognizing the need to add dimensions of fire resilience to their programs after being exposed to ideas in the other communities and landscapes through the series of learning exchanges hosted by FAC Net. When asked how their work has changed as a result of the exchanges, one member explained, “It changed who I was talking to, it changed how I developed partnerships, it changed how I worked with our steering committee – I just changed every part of how I was operating in my silo.”
Using a network to spread ideas and models for fire adaptation is impacting fire management at a systems level. One REM participant described how they are “working with communities in Washington and Oregon, I’m hearing what they’re saying, and what [the network is] saying, too. When I go to [a stakeholder or policy maker] I’m able to take those messages and move them up the chain.” Specifically, members are using the network to support changes to state strategic plans and air quality regulations.
Conclusions
We gained an incredible amount of insight through this three-part evaluation. Detailed information about member connections discovered via SNA has led us to netweave in new ways. We’ve also recognized the need to expand FAC Net’s core and periphery relationships—building on the foundation we’ve developed. Positive feedback from the network health scorecard points to aspects of the network’s operations and design that are working well for members, and validates the idea that FAC Net is a well-functioning system. Stories gathered in REM illustrate how FAC Net’s culture of adaptability, reciprocity, learning and trust are valued and shared by members. And how these values, and our connectivity, are allowing us to change members’ individual practices, spread those ideas and, ultimately, inform and drive systems change.
Originally published at FireAdaptedNetwork.org