By Juliana Bedoya Carmona, Monitoring and Evaluation intern
I have recently joined the FrontlineSMS team in Nairobi as part of an ongoing Monitoring and Evaluation Project rolled out by FrontlineSMS in collaboration with Tufts University Team; Bronwyn Cook, Shuvam Dutta, Amanda Meng and Julie Younes, who are also part of FrontlineSMS Heroes. So far, my experience in Nairobi has been very enriching. Getting to know the FrontlineSMS Nairobi team, meeting FrontlineSMS users and finding out about other M4D and ICT4D start-ups working out of Nairobi’s iHub, has been a very exciting and valuable learning opportunity.
Since December 2011, FrontlineSMS, in collaboration with a multidisciplinary group of graduate students at Tufts University (from the Fletcher School of Law and Diplomacy and the Department of Urban and Environmental Policy Planning) and under the direction of Dr. Jenny C. Aker, has engaged in the development of a new Monitoring and Evaluation framework. As part of this process, I have recently joined the FrontlineSMS team in Nairobi to roll out a test phase of the M&E framework that includes working closely with user organizations in the field, receiving their feedback and analyzing preliminary data provided by them to further improve the framework as it currently stands. The objective of the M&E framework is to capture data that illustrates what type of impact FrontlineSMS may have in user organizations' everyday activities in terms of cost, time and communications efficiencies savings. By developing and testing this framework we aim to better understand and improve the ways in which FrontlineSMS may effectively advance towards achieving its mission; that of lowering barriers to transformative social change by using mobile technologies.
NGOs and other non-profit organizations have used FrontlineSMS in a wide variety of contexts and with myriad goals in mind. From real time malaria diagnosis in Cambodia to tackling violence against children in Benin, FrontlineSMS is being used in over 80 countries to lower communication barriers and facilitate social change around the world. But how exactly does FrontlineSMS transform everyday operations of user organizations and what are the dimensions of these changes?
Measuring the impact of information and communication technologies for development (ICT4D) is drawing increasing attention as more organizations engage with technologies to help them press forward their organizational goals. There are, however, specific challenges linked to monitoring and evaluating efforts in this particular area. Often organizations implementing mobile technology tools such as FrontlineSMS in their operations do not set out from the start to record changes related to the incorporation of this technology. From the viewpoint of organizations providing the technologies, such as FrontlineSMS, tracking performance of their product often proves difficult as they do not have direct access to the data showing efficiencies or changes brought about by the implementation of a given technology. Given that FrontlineSMS is open source software it is also not easy to keep track of the innovative ways in which user organizations are implementing the software in their everyday operations, either as standalone or in combination with other applications. Furthermore, gathering enough data in order to draw confident enough conclusions depends on the number of organizations willing to participate in the M&E framework, fill out surveys and provide feedback that helps tailor the evaluation instruments and selected indicators. As a result, the process of developing an M&E framework for FrontlineSMS has been a gradual and participatory one involving FrontlineSMS staff members, researchers and finally user organizations.
FrontlineSMS has been keen to listen to user organization's experiences of using the software through its annual user survey, the online community space and the development of case studies about specific organizations. The new M&E framework builds on these tools and incorporates additional ones to help draw a more complete picture of the changes that organizations notice after implementing FrontlineSMS, and how these compare among different user organizations. The elements of this new M&E framework include a logical framework, a new survey and the development of new case studies that provide additional qualitative evidence to help refine the tools that have been produced so far.
In the next couple of days we are looking to finalize a list of user organizations willing to partner with FrontlineSMS in this effort. By including user organizations in the development and testing of the first phase of the M&E framework we aim to improve the design of the system in order to capture as much rich data as possible, which will then be useful to the FrontlineSMS team and our users. We anticipate that the M&E framework as well as its results will contribute to improved understanding of the usability, benefits and challenges of implementing FrontlineSMS software; providing a clear picture of the types of costs, time and communication efficiencies that could be achieved by using FrontlineSMS; and offer a basic framework for user organizations to track their own communications, cost and time management performance in those areas in which they implement FrontlineSMS.
If you would like to find out more about our M&E framework please contact us at info@frontlinesms.com . We are particularly keen to hear from new and existing users of our software eager to measure the impact FrontlineSMS is making to their work.
Our first visit was to Josana Academy, where we met the head teacher, secretary, bursar, and IT support person. The first step was to install PaymentView on the secretary’s computer, as she will be the primary user. Next, we trained the secretary, bursar, and IT support person on how to use the software. Josana Academy will be using PaymentView to enable easier processing of fee payments made via mobile money. Josana is a private primary school, so fees are collected every term. Currently, parents who live far from school and/or cannot easily access a bank branch will ask the school if they can pay via M-Pesa. These payments are either received by the secretary or bursar, or they are received by a child’s classroom teacher. Once a payment is received, the receiver must go into town to cash out and then deposit the money into the bank. PaymentView will streamline this process by enabling all payments to go to one, centralized place. This new process minimizes the number of trips to town made by the secretary or bursar, and prevents teachers from having to make trips to M-Pesa agents for withdrawal.