Villgro Blog: ANDE Impact Report

In an earlier post I had pointed out some weaknesses in the research methodologies of the Ashoka and Beyond Profit surveys of social entrepreneurs. At around the same time that the results of these two surveys were published, ANDE also published its own impact report. ANDE is the Aspen Network of Development Entrepreneurs. How does their report compare to the Ashoka and Beyond Profit surveys?

In brief, the ANDE report is more transparent about how it arrived at its results than the Ashoka and Beyond Profit surveys. However, there is some variability in the completeness and clarity with which the data is presented.

Although ANDE’s report describes itself as an “impact report”, it is really only two sections that are about ANDE’s work. In addition, there is no mention of the methodology used to collect the data in these two sections. Despite the strengths of the report, this is a major weakness.

The two sections that are on ANDE’s work are titled, “ANDE’s Role and the Impact of Our Members”, and “ANDE’s Efforts To Grow The Sector To Scale”. Of these, the latter section consists of straightforward reporting on ANDE’s activities over the year. Therefore I will focus only on the section titled, “ANDE’s Role and the Impact of Our Members”.

In this section, it is Figures 9-14 that I would like to comment on. All figures report results in terms of percentages. In my earlier post I had said that the problem with using percentages alone is that we have no way of knowing if these results are due to chance, because these results were not tested using a test like the chi-square or t-test. However, I realized that I need to explain this further.

Chi-squares or t-tests are needed if you are collecting data from a sample of respondents, and using your results to generalize about the larger population that they (supposedly) represent. Using percentages is not a problem if you have collected data on the entire universe that you are studying. Look at Figure 13 of the ANDE report, for example. The figure is titled, “How ANDE Members Fund SGBs’ Financial Needs”. We are told earlier in the text that ANDE members who invest in SGBs (Small and Growing Businesses) manage 51 funds (23/24). The N=51 at the bottom of Figure 13 tells us that data from all 51 funds is represented in this figure.

Similarly, Figure 11 is also quite clear. It shows how many ANDE member funds have a target return range of 0-5%, 5-20% and above 20%. Here, as in Figure 13, the universe should be 51, as that is the number of funds that ANDE members investing in SGBs manage. However, it is explained that for this figure N=48, as three ANDE member funds did not provide their target IRR range. The only flaw in this figure is its titling. It is titled, “ANDE intermediary target benefit: Percent of ANDE member funds with target IRR range”. This suggests that respondents were asked a yes/no question, such as, “Do you have a target IRR range?” Instead, respondents were probably asked a question like, “What is your target IRR range?” and their responses indicate the spread of ranges. In addition, IRR is not defined in the figure, nor is there a glossary.

Other figures are more ambiguous. There is no N given for Figure 10. Since the title of the figure is, “ANDE intermediary target size: Percent of ANDE member funds with target average investment size”, should we assume that N=51? The title also suggests a yes/no question as with Figure 11. Figures 12 and 14 both say that N=70, and includes both funds and capacity building providers. Yet the term “capacity building providers” is not explained anywhere else in this section.

The text in this section is well-supported by footnotes, and doesn’t leave room for misinterpretation. Where the report says that, “ANDE members have made 2,499 investments in SGBs totaling $830 million (26/27)”, we can see from footnote 15 that this information was collected from all 51 ANDE member funds. Where it says that, “33 ANDE members spent $96.8 million on technical-assistance activities (26/27)”, footnote 17 tells us that data was not available for the remaining ANDE members. And the $1.7 billion in additional funding that ANDE portfolio companies have secured is, as footnote 19 tells us, based on reporting by 21 funds.

It is important to note that, as in some of the above examples, ANDE has most likely underreported their results rather than extrapolate where data is not available. This is to be appreciated. The one exception I found, where the footnote did not explain the text well, was 16. If you read the text and the footnote together, it says, “Among those funds that reported historical investment-size information for these past investments, 96 percent of the total number of investments made were investments under $2 million (26/27)”, excluding one ANDE member fund representing 450 investments that did not report quantity under $2 million. We don’t know what the total number of respondents was (those that reported historical investment-size information), and we also don’t know why one fund was excluded.