James Tooley
ABSTRACT
Muralidharan and Sundararaman report a randomised controlled trial of a school voucher experiment in Andhra Pradesh, India. The headline findings are that there are no significant academic differences between voucher winners and losers in Telugu, mathematics, English, and science/social studies, although because the private schools appear to use time more efficiently, they are also able to teach Hindi (the national language). The average per capita cost in private schools is less than a third of that in public schools. So while private schools are more efficient, they are not necessarily leading to higher standards. There are two types of private school in the experiment, English and Telugu medium. Since tests in non-language subjects were conducted in a different language for children in public and English-medium private schools, the results in mathematics and science/social studies are difficult to interpret. There are suggestive comparisons between children in Telugu-medium private and public schools, where children took tests in the same language (and were also not subject to disruption in medium of instruction), which show that students in private schools outperform those in public in all subjects. This suggests that giving children access to private schools through vouchers could be a very important policy reform.
Introduction
A recent DFID-commissioned report (Day Ashley et al., 2014) revealed areas of consensus and controversy (see Tooley & Longfield, 2015) around the phenomenon of low-cost private schools serving poor families in developing countries. One area illuminated by the report concerned learning outcomes. The headline finding was that ‘Pupils attending private school tend to achieve better learning outcomes than pupils in state [i.e. public1] schools’ (Day Ashley et al., 2014, p. 15). However, this finding is tempered by the caveat that there aren’t many good studies available (they point to only three of ‘high quality’); many have the shortcoming of not being able sufficiently to control for background and possible missing variables and selectivity biases. Another set of authors concur:
There is very little rigorous empirical evidence on the relative effectiveness of private and public schools in low-income countries. Non-experimental studies have used several approaches to address identification challenges and have typically found that private school students have higher test scores, but they have not been able to rule out the concern that these estimates are confounded by selection and omitted variables. (Muralidharan & Sundararaman, 2015, p. 1013)
These authors, in a paper that emerged since the publication of the DFID-commissioned report (so not included as evidence), set out to fill this research lacuna by presenting experimental evidence from a school choice experiment in Andhra Pradesh,2 India (Muralidharan & Sundararaman, 2015, p. 1013). The study, published in Quarterly Journal of Economics, features a randomised controlled trial, with a ‘unique two-stage randomization’ process, held up as the ‘gold standard’ of research in this area (see e.g. Mantri & Gupta, 2014). The headline findings of the research show that children using vouchers in private schools attain the same level of academic achievement as those in public schools but at a third of the cost. Furthermore, there are no spillover effects on children in public schools whose parents didn’t apply for a voucher or those already in private schools.
These findings have been widely—and disparately—reported as heralding a conclusion to the debate about the relative merits of private and public schooling: For example, Karopady (2014) asserts ‘The empirical evidence is increasingly pointing towards private schools not being able to add value as compared to government schools’ (p. 51). The Times of India suggested:
The findings of the Andhra Pradesh School Choice research aren’t encouraging for voucher systems supporters. Private school kids performed better than government school ones in only the first year; in subsequent years, government ones performed just as well. … The findings dispel a popular myth that private schools lead to better learning. (Chowdhury, 2015)
Meanwhile, supporters of School Choice rallied to private schools’ defence: writing in response to the Times of India article, one of the foremost defenders of vouchers comments:
If I were to write the title for the Times of India story, it would be: at three times the cost, government schools are no better than private schools. The Times of India headline is: private schools are not adding value. you be the judge! (Shah, 2015)
Note that the research findings are accepted by both critics and supporters of vouchers alike; this paper takes a different approach. While congratulating the authors on their powerful research design, it suggests that there is an important challenge in the research implementation which calls into question the headline results: the challenge is that many voucher-winning students took the tests for non-language subjects in a different language (English) than voucher-losing students in the public schools (Telugu). (They also switched their medium of instruction to English-medium schools, and were therefore subject to disruption in their medium of instruction.) Fortunately, the research paper does give suggestive indications of what a comparison would be like when children in public and private schools did take the same tests. This shows a large and statistically significant private school advantage in achievement, as well as the widely accepted cost-effectiveness.
In this paper, I first briefly summarise the methods and findings of the research paper. Second, I point to the key problem with the research implementation which calls these findings into question. Third, I discuss the suggestive results which do allow direct comparisons between children in public and private schools. Fourth, I note other less central issues which may nonetheless be of interest, especially to those trying to design effective comparisons between public and private schools. Finally, I discuss the authors’ policy prescriptions and add some reflections of my own.
Summary of method and findings
The research examined the impact of the Andhra Pradesh School Choice Project, which conducted a voucher experiment in five districts of rural Andhra Pradesh, south-central India, funded by the Legatum Foundation, the UK’s DFID and the World Bank. The experiment featured eight stages:
- Villages which were in the same districts as those used in the larger Andhra Pradesh randomised Evaluation Studies (although in different divisions to avoid any connection with schools in the other studies) and in which there was at least one recognised private school were selected for the study—there were 180 such villages.
- Baseline tests in Telugu and mathematics were given to all students in two cohorts: those attending the final year of pre-school and those attending grade 1 of primary school, in both public and private schools in these 180 villages. Tests were conducted in March–April 2008, i.e. at the end of the school year.
- Parents of students in public schools in all of these 180 villages were invited to apply for a voucher.
- The first part of the two-stage lottery procedure took place: 90 out of the 180 villages were randomly assigned to be ‘voucher villages’ (Muralidharan & Sundararaman, 2015, p. 1023), while the remaining villages would continue as before.
- The second part of the two-stage lottery took place: within the voucher villages, children from public schools were randomly selected to be voucher recipients. Invitations were given to parents of these randomly selected students to apply for a voucher so that they could attend private school. (Thus, importantly, stages 3–5 led to their being ‘two lottery-based comparison groups—those who did not get the voucher due to their village not being selected for the program’ and ‘those who did not get the voucher due to losing the individual level lottery conducted within voucher villages’ [Muralidharan & Sundararaman, 2015, p. 1023]).
- Recognised private schools in the villages were invited to participate in the voucher programme.
- Tests in Telugu, mathematics and English were conducted at the end of two and four years, while tests in science and social studies (called ‘EvS’ in the paper) and Hindi were given after four years.
- Household surveys were conducted once every year in a representative sample of households of the different sets of students, and data were collected from unannounced school visits conducted once a year in all schools in the 180 project villages.
Overall, 23% of government school children in the voucher villages were reassigned to private schools. The baseline tests (conducted when students were at the end of pre-school or at the end of primary grade 1) showed a highly significant difference of 0.65σ in favour of children in private over public schools (Muralidharan & Sundararaman, 2015, p. 1039 and Table II, p. 1027). For children receiving vouchers over the four years of the experiment, however, the conclusion is that differences in favour of private schools are not to do with features of private schools:
After two and four years of the program, we find no difference between test scores of lottery winners and losers on Telugu (native language), math, English, and science/social studies, suggesting that the large cross-sectional differences in test scores across public and private schools mostly reflect omitted variables. (Muralidharan & Sundararaman, 2015, p. 1012)
However, apparently because they are able to more efficiently allocate time taught on other subjects, private schools in addition teach, inter alia, Hindi, whereas the public schools do not; unsurprisingly lottery-winning voucher students do better at Hindi then than those left in the public schools. Moreover, ‘the mean cost per student in the private schools … was less than a third of the cost in public schools’ (Muralidharan & Sundararaman, 2015, p. 1012). Hence the final conclusion:
private schools in this setting deliver slightly better test score gains than their public counterparts (better on Hindi and same in other subjects), and do so at a substantially lower cost per student. (Muralidharan & Sundararaman, 2015, p. 1012)
This is the headline conclusion that as we noted above has been accepted by voucher proponents, as well as critics. We shall explore in the next section why this headline conclusion appears insupportable because some tests used were different for public and private school students.
The fundamental problem
Researchers wanting to compare achievements in public and private schools in India have long faced a dilemma: what language should be used for testing children in mathematics and other non-language subjects? (see e.g. Bashir, 1997; Dixon, 2003; Kingdon,1996). The dilemma arises because of differing mediums of instruction in public and private schools. Typically, the medium of instruction in public schools is a regional language, thus Telugu in rural Andhra Pradesh. Private schools on the other hand often purport to be English medium (although the reality is more complex, see below), but others are also (in Andhra Pradesh) Telugu medium. Hence, if a mathematics (or other non-language subject) test is given to children with written instructions in English, this may seem to privilege those from the English-medium private schools, whereas giving instructions in Telugu might appear to be biased against those used to receiving and understanding instructions in English.
Two common solutions have arisen to this dilemma:
- Ensure that mathematics (and other non-language subject) tests are word-free, e.g. arithmetic operations only and/or wordless cognitive puzzles.
- Ensure that the instructions given in mathematics (and other non-language subjects) tests are in both languages on the same paper, to ensure that students can choose which language to use for instructions on how to address each individual question.
Even in the second case, problems of translation will occur—it’s hard for researchers (even those familiar with both languages) to be sure that word questions in mathematics have been identically translated, as it is known that even small changes in words (as well as other features of questions) in mathematics questions can drastically alter student success rates (see e.g. APu, 1988). For this reason, the first solution may be the most preferable even though it does limit the type of mathematical questions that can be asked, although the second is also used.
Nowhere in the paper is it explicitly mentioned, but as the problem is widely known, the assumption must be that the researchers solved this dilemma in one of these two standard ways. Unfortunately, they did not.
Recently, I asked one of the project researchers (acknowledged in Muralidharan and Sundararaman [2015, p. 1011]), to conduct tests to compare the low-cost private schools where I was working, with public schools, in Hyderabad, Andhra Pradesh. His team used different tests in mathematics for public and private schools, with instructions in English for the private schools (which were all, ostensibly at least, English medium) and Telugu for the public. He assured me that this was the method used in the Andhra Pradesh School Choice Project. This was confirmed by one of the authors: while the protocol was for schools and/ or pupils to be given a choice of which paper to use, the language used in the mathematics (and other non-language) tests ‘tended to follow the medium [of instruction] of the school, with English-medium private school students taking the test in English and Telugu-medium students taking the test in Telugu (the split was roughly 50% each)’ (Karthik Muralidharan, personal communication; of the private schools, 50.4% were English medium [Karopady, 2014, p. 49]). Note, importantly, that this is not the same as the second solution to the language dilemma given above. There both languages are given on the same paper, so that all students still take the same test, but can choose which language to read. In the case in question, students took in effect different tests in mathematics and other non-language subjects.
The aim of creating a randomised Controlled Trial is as far as possible to ensure that participants in treatment and control groups are treated in exactly the same way apart from the unique factor introduced as the intervention—in this case school vouchers. This study it appears has treated the treatment and control groups differently by using two different sorts of tests for mathematics and EvS: one in English, the other in Telugu. Even if it wasn’t obvious in what ways this difference in treatment could lead to bias for one or other groups of participants, one would be justified calling into question the results. However, in this case, one can see clearly how the different tests could cause very serious bias.
If it was a comparison between performance in public and private schools in well-heeled urban communities it may be thought that it was less of a problem: the children in English- medium schools are being taught in English, so why shouldn’t they be given tests in English? However, in poorer rural (or urban slum) areas of India, the ‘English medium’ appellation carried by low-cost private schools is typically more of an aspiration, at least in the lower grades, than a reality. As Karopady, from Azim Premji University, who had been closely involved with the project, (acknowledged for his ‘constant support’ [Muralidharan & Sundararaman, 2015, p. 1011]), put it:
The medium of instruction is as claimed by the school authorities. In the rural setting, while these schools could have more transactions in English, they are some distance from being truly English medium. (Karopady, 2014, footnote 6, p. 52, emphasis added)
This is agreed. I have noted elsewhere, for instance, how it is an ‘oft-repeated criticism’ of low-cost private schools that they are ‘English medium in name only’ (Tooley, 2009, p. 179); I also reported that ‘Observations in classrooms … suggest that schools describing themselves as English-medium use English only textbooks … with teachers offering a mixture of English and Urdu/Telugu to support their use …’ (Tooley, Dixon, & Gomathi, 2007). The low- cost English-medium schools in fact operate as hybrid schools—teaching in the mother tongue in the lower grades, often using English textbooks translated by teachers into the mother tongue, with the aspiration of bringing everyone up to speed in English by higher grades.
Hence even in a simple comparison between public and private schools in rural areas, it would be unfair to give tests with English written instructions to children in private schools (supposedly English medium but in fact teaching in Telugu in the lower grades), as this would penalise them against those being given tests with Telugu instructions. In this particular voucher experiment, the situation appears even more difficult: ‘Overall, the students who applied for and accepted the voucher had lower baseline test scores, suggesting that students with lower test scores were more likely to leave the public schools if given the opportunity to do so’ (Muralidharan & Sundararaman, 2015, p. 1028). So children switching from Telugu-medium public to English-medium private schools were likely to have been of low achievement levels; for them, trying to figure out mathematics questions in English may have presented huge difficulties.
To illustrate the kind of problem faced by these voucher students when doing tests after being moved to English-language schools, I’ve translated three of the questions from the mathematics test into Telugu. (I’m assuming not many readers understand Telugu; if they do, they’ll need to translate into a language they don’t understand.) readers will have no difficulty with questions like this one:
క్రింది పరిష్కరించండి
5.57+3 = _________
Although the instruction is in Telugu, it is in effect repeated in the mathematical language too, so as long as we know how to add together two numbers (whilst taking into account place value) we can get this one correct, irrespective of the language of the test (the Telugu says ‘Solve the following’).
But this next question is entirely different:
ఏ అంకెల సంఖ్య 2345 లో వంద స్థానంలో ఉంది ?
Without knowing Telugu it is simply impossible to answer, however good we are at mathematics. (The question is: Which digit is in the hundred’s place in the number 2345?)
Similarly, we will not be able to solve this question, unless we are fluent in Telugu:
వైశాలి ఒక పెన్సిల్ విలువ రూ కొనుగోలు కోరుకుంటున్నారు. 4. ఎన్ని 50 పైసల నాణేలు ఆమె
పెన్సిల్ కొనుగోలు అవసరం ఉంటుంది ?
(vaishali wants to buy a pencil worth rs.4. How many 50 paise coins will she require to buy the pencil?)
The point is that the difficulties we have with these last two questions illustrate precisely the nature of the problem faced by voucher children moved to an English-medium private school. However good they are at mathematics, they will not get these answers correct, except of course through lucky guesses.
Of course, the argument could be framed in the opposite direction—that testing the children in English-medium private schools in Telugu would have been unfair, especially in the later years of the experiment, as by then voucher children’s English may have improved because of their greater exposure to the language. However, it must not be assumed that the language children will learn in English lessons is the same language they will need in mathematics, or that there is equal degree of English-language immersion in language and non-language subjects, especially in the grades tested. It is plausible, for example, that mathematics’ teachers were less fluent in English than the language subject teachers, and so placed a greater emphasis on teaching in Telugu than language teachers. Moreover, it is well known that the language used in mathematics lessons and tests is often very different from that used in English lessons. There are also problems of unknown or misunderstood vocabulary, with ‘different meanings in everyday usage, as with even, odd, and function’ (Math Solutions, 2009, p. 5); ‘How many are left?’ does not signify a directional question at all. A further difficulty is that the same mathematical concept can be conveyed using many different words—e.g. ‘add’, ‘and’, ‘plus’, ‘sum’, ‘combine’. There are confusing mathematical and non-mathematical homonyms, such as ‘sum/some’ and ‘whole/hole’. And mathematics questions often come ‘embedded in language that makes the problem unclear or difficult to comprehend’ (p. 5). Overall, those learning English as a second (or greater) language ‘typically experience difficulty understanding and therefore solving word problems, and this difficulty increases in later grades of elementary school as the word problems become more linguistically and conceptually complex’ (p. 6), out of synch with the language taught in the corresponding grade English lessons.
The key points are, first, the only fair way of assessing the students in different language-medium schools would be to follow one of the two methods outlined above, using word-free tests, or using tests with both languages translated side-by-side. As this was not done, secondly, we simply do not know what the impact of having used these different tests will be on student performance. This therefore leads to the following re-phrasing of the headline findings of the Andhra Pradesh School Choice Project, given that the results are for both English- and Telugu-medium private schools combined, compared to Telugu- medium only public schools:
- In Telugu, the regional language, there is no significant difference between achievement of those in public schools and those receiving vouchers to attend private schools. However, private schools spend significantly less time on Telugu so they are much more efficient than public schools.
In mathematics and EvS, we do not know what advantage or otherwise the private or public schools have. As children took different tests, there is no basis for comparison. - In English, the international language, children with vouchers in private schools perform slightly better than those in public schools.
- In Hindi, the national language, children with vouchers in private schools perform better than those in public schools. (This is not surprising, however, given that public schools do not teach Hindi.)
That is, in the five subjects tested in year 4, voucher children in private schools perform (at least slightly) better in two (international and national languages) and the same in one (regional language); in the other two non-language subjects, we cannot tell what the difference is.
In fact, the finding in English may also not be robust, because of further difficulties with the English test. The English tests (as well as the other subjects) ‘were carefully designed to assess the common curriculum in government and private schools so as to ensure that there was genuine comparability’(Karopady, 2014, p. 49). But this presents an immediate difficulty for English as opposed to other subjects: English is taught in private schools from Class I whereas in public schools only from Class III (p. 50). The same tests administered in public and private schools would then bring in problems of potential bias: if the tests covered what the private school children had covered, then they would be unfair to the public school children. If they measure what public schools have covered by Class III, then they would be too easy for the private school children, leading to a ceiling effect on what they were able to demonstrate. It turns out that there may well have been such a ceiling effect. The researchers, aware of the low level of achievement of all students, designed the tests to include items from the tested grade and several lower grades too (personal communication, Karthik Muralidharan). This means that the English results are likely to underestimate the true private school effect.
It is obviously disappointing that we can’t say anything about mathematics (or science and social studies). However, as roughly half the students in private schools (those in Telugu- medium private schools), did take the same mathematics and science/social studies tests as those in public schools, can’t we look at the results for these children to get a fairer comparison between public and private? Helpfully the researchers did explicitly compare these groups, with revealing results.
Suggestive results comparing like with like
The researchers did disaggregate results for the Telugu- and English-medium private schools, and found rather interesting results. It is important to stress that the discussion in this section, following the caveats in Muralidharan and Sundararaman (2015, pp. 1047–1055), is suggestive only, (‘some suggestive patterns emerge in the results’ [p. 1054]), for two major reasons: first, the standard errors, when following a ‘conservative Iv [instrumental variable] strategy’ are ‘too large for meaningful inference’ (p. 1054); moreover, even ‘with a precise Iv estimate’, the researchers note that the medium of instruction of the school ‘is correlated with other school characteristics’ (p. 1054). However, reinforcing the above discussion that the different tests used may have distorted the results, they note that in general English-medium schools ‘have superior indicators of school quality—including facilities; teacher experience, qualifications, and salary; and annual fees charged per child’ (p. 1054). It is counter-intuitive to expect student outcomes in these better equipped schools to be lower than in schools with fewer advantages.
The results disaggregating children in English-medium and Telugu-medium private schools are as follows:
At the end of four years of the voucher program, we find that the causal impact of attend- ing an English-medium private school varies sharply by subject, with students doing worse (than staying in the public school) in Telugu, math and EvS but much better in English and Hindi. The mean impact across subjects is positive (0.22σ) but not significant. (Muralidharan & Sundararaman, 2015, p. 1051)
We must reinforce again the point raised above that the mathematics and EvS scores here are called into question. Because the children took different tests in the English-medium private and Telugu-medium public schools, we simply do not know about the relative achievement of children in public and private in mathematics and EvS.
However,
On the other hand, the estimated impact of attending a Telugu-medium private school is positive for every subject, and the mean impact across subjects is positive (0.53σ) and significant. (Muralidharan & Sundararaman, 2015, p. 1051)
Because in the Telugu-medium private schools all tests were the same as those in the public schools, and there was no confounding variable introduced of medium of instruction, it is fair (although only suggestive, for the reasons given above) to compare children in Telugu- medium private schools with those in public schools. Table 1 (simplifying the researchers’ Table x [Muralidharan & Sundararaman, 2015, p. 1052]), shows the results of comparing like with like.
The results can be summarised as follows:
- In year 2, estimated score differences between private and public schools are positive in favour of those having vouchers for all subjects apart from Telugu.
- By year 4, estimated score differences are positive for every subject, and the mean impact when subjects are combined is large and positive (0.53 standard deviations) and statistically significant. Importantly, this is not simply the effect of Hindi distorting the results: combining mathematics and EvS also gives a large (0.50 standard deviations), positive, and statistically significant difference, albeit at the 10% level (Muralidharan & Sundararaman, 2015, p. 1052, Table x).
Children with vouchers in the private schools outperformed those in the public schools in all subjects after four years of the voucher programme; the combined result shows a large, statistically significant difference in favour of private schools. This is a hugely positive, albeit suggestive, finding for the voucher debate.
The authors see the significance of these findings as follows: ‘students who switched from attending a public school to a Telugu-medium private school did better than those attending an English-medium one (especially on non-language subjects)’. This suggests for the researchers that ‘private schools may have been even more effective when students did not experience the disruption of changing their medium of instruction’ (Muralidharan & Sundararaman, 2015, p. 1015).
Ceteris paribus, this might be a useful explanation to explore in terms of ‘education psychology literature, which suggests that first-generation learners may be better off being taught in their native language, which can be reinforced at home’ (Muralidharan & Sundararaman, 2015, p. 1055). But other things were not equal; as already discussed, children in English-medium private schools were given different tests to those in the Telugu-medium schools, so we simply do not know what their relative performance actually was. As indicated above, there may not be a major difference between the language of instruction for younger children in the (supposedly) English-medium private schools and the Telugu-medium private and public schools, so testing them in different languages is likely to be unfair. All are likely to be taught, at least predominantly, in their mother tongue, perhaps in one case using English-language textbooks. Hence, because of the research implementation using different tests, we are not able to confirm whether or not the school language of instruction actually impacts on student performance given these results.
Miscellany of minor issues
Some other issues may be worth highlighting, in lesser detail (given space constraints), each of which illustrates the difficulties that need to be overcome in order to produce ‘gold standard’ research in this area.
Pupil attrition
One of the major difficulties that arose in this research—and which perhaps could not reasonably have been anticipated—is pupil attrition. The researchers write: ‘We attempted to administer the written tests to the full set of students’ (Muralidharan & Sundararaman, 2015, p. 1029), but initially failed to do so because of high levels of student attrition (of around 40%), most of which was caused by ‘students who had migrated and could not be found, as opposed to students still attending schools but not present for testing’ (footnote 18, p. 1029). Given this, the researchers then had to mount an ‘intense effort … to track down all the students who had applied for the voucher’. Once done, they had to conduct an additional round of testing in each village outside school hours. The same process—this time not unexpectedly—was required in year 4 of the scheme (footnote 18, p. 1029).
Class size
In the research, ‘a large fraction (23%) of public school students moved out to private schools’ (Muralidharan & Sundararaman, 2015, p. 1045). Prima facie, then, one would assume as a result class sizes had decreased in public schools and increased in private schools, which could have led to bias against private schools and certainly would have violated the principles of randomised Controlled Trials, of keeping as much as possible the same for both control and treatment groups. However, the authors ‘verify that being in treatment villages does not change the average of several key school characteristics between treatment and control villages over the course of the study (results available on request)’ (footnote 19, p. 1030). Two reasons are given for the lack of private school class size increase: first, private schools ‘used the additional resources provide by the voucher payments to … keep enrolments constant’ (footnote 19, p. 1030). Anecdotally, I’ve rarely heard of a private proprietor who didn’t want to get more income, so it seems odd that any would turn pupils away. Second, the private schools hired ‘enough staff so that average characteristics (such as class size) did not change on average’. This also doesn’t gel with my experience, as typically low-cost private schools in villages do not operate at full capacity. So if a school gets more children (on vouchers) then they would fill up spaces rather than employ extra teachers to create new classes (especially as it is stipulated that no class had more than 25% of its enrolment as voucher kids).
KG or not?
Did the public school children attend kindergarten or not? There appears to be a contradiction in the paper here. On the one hand, the authors write:
a major limitation in the cross-sectional comparisons is that private school students typically have two years of pre-school education (nursery and kindergarten) compared to public school students (who typically start in the first grade). Thus, comparisons of test score levels at a given primary school grade confound the effectiveness of private schools and the total years of schooling. (Muralidharan & Sundararaman, 2015, footnote 9, p. 1018, emphasis added)
In passing we can note that it doesn’t seem an impossible task to control for this variable (ask the children and/or the families how many years of schooling they have had, including pre-schooling). But more importantly, on the other hand we are also told that for children in both private and public school samples, ‘The cohorts covered were students attending kindergarten and grade 1 in the previous school year (2007–08)’ (footnote 13, p. 1021). In other words all children tested, both public and private, were in school, either anganwadi (public pre-school) or private kindergarten. This is confirmed in other reports of the same research: ‘A baseline test … was administered during March–April 2008 to children in angan- wadi or KG’ (Karopady, 2014, p. 49). Indeed, the experiment ‘was intended for students who were studying in anganwadis or in KG in the academic year 2007–08 and who had intended to study in government schools’ (p. 49, emphasis added). Hence, whether it is true or not, as the authors claim, that in general children in public schools don’t attend pre-school, for the chosen sample this was not true—they were actually selected whilst in a public or private pre-school.
Conclusion: discussion and implications
While the recent DFID-commissioned ‘rigorous literature review’ (Day Ashley et al., 2014) reported that children in private outperform those in public schools, this conclusion is based on findings from a relatively small number of studies, none of which has been experimentally-based. One recently published study aimed to fill this lacuna. Muralidharan and Sundararaman (2015) conducted an experiment to ascertain the impact of ‘School Choice’ in Andhra Pradesh, India. They used the ‘gold standard’ research method of randomised Controlled Trials (rCTs), with an innovative two-stage randomisation procedure. The headline results reportedly showed that children given vouchers to attend private schools did not in general outperform those left in public schools. However, private schools delivered the same results as public schools at a third of the cost, so reforms to allow vouchers in private schools will be more cost effective than public schools.
However, it turns out that the same tests were not used for children in public and private schools, violating a key principle of randomised Controlled Trials, that the different groups should be treated in exactly the same way. This means that the headline results are called into question: roughly half of the private schools (‘English medium’) used non-language tests with English instructions, whereas the public schools and the remaining private schools (‘Telugu medium’) were given tests with instructions in Telugu. This means that like was not compared with like, and so we are unable to say what the differences are between public and private in non-language subjects.
Fortunately, the researchers were able to disaggregate the results in English-medium and Telugu-medium private schools, although these results are subject to important caveats so are suggestive only. Comparing the results for children in Telugu-medium private schools with those in public schools, where exactly the same tests had been given, children given vouchers to attend private schools achieved better in all subjects (Telugu, mathematics, English, science/social studies and Hindi) than those left in public schools. In other words, these findings are suggestive that voucher children in private schools significantly outperform those in public schools, for a fraction of the cost.
The issue of cost needs to be addressed. The private schools in the sample had an average annual cost (which presumably is more or less equivalent to the average annual fee) of around rs.1849 per child, or rs.185 per month over 10 months (about USD 4 per month using historical exchange rates) (Muralidharan & Sundararaman, 2015, Table III, p. 1031). This was significantly lower than the average annual cost of public schools (at rs.8390 per child). Certainly, on average, schools in the sample are low-cost (see Tooley and Longfield [2016] for a discussion of ‘low-cost’).
One of the ways in which such schools are able to keep costs low is by paying teachers lower wages than in government schools (Muralidharan & Sundararaman, 2015, p. 1032), which some view as exploitative (Day Ashley et al., 2014, pp. 21–22). Indeed, in India this comparative advantage of low-cost private schools prima facie appeared to be outlawed under the ‘right of Children to Free and Compulsory Education Act’(2009), usually known as the right to Education Act (RTE). under RTE, regulations for private schools have been created which are focused on input measures, including teacher qualifications and salaries. As it stands, this would seem to make the operation of many low-cost private schools illegal, and thousands of unregistered schools have been closed as a result (see e.g. Francis, 2014). However, the prima facie emphasised above is important. under the Indian constitution, although states are obliged to implement the Act, they are permitted flexibility in interpreting and modifying the regulations, provided that the basic provisions are not violated. At least two states, Gujarat and rajasthan, have sig- nificantly modified the regulations, allowing in some cases schools to be judged in large part on their student outcomes rather than on inputs (see Gujarat Government, 2012, Appendix 1). In these states, and if this approach to the regulations became more widely accepted, then it would appear that low-cost private schools could continue as before in the way they pay teach- ers, provided of course that they achieve high enough student outcomes.
Muralidharan and Sundararaman (2015) note some policy prescriptions of their research findings:
Our results on private school productivity suggest that it may be possible to substantially increase human capital formation in developing countries like India by making more use of private provision in the delivery of education. (p. 1058)
However, they note important caveats to this, including:
there may be a trade-off between a libertarian approach to school choice that believes that parents will make optimal schooling choices for their children and a paternalistic one that believes that parents (especially poor and uneducated ones) may make misguided evaluations of school quality based on visible factors that may not contribute to more effective learning. (pp. 1061–1062).
They continue:
While we find that private schools are much more productive than public schools from the perspective of a social planner, it is not obvious that they represent a better value for the marginal parent who is paying for private schools over a free public school. Since test scores did not improve in math and Telugu, the marginal parent would have to place a high value on Hindi scores to justify paying for the typical private school in our sample. Although we cannot rule out this possibility (or that parents valued other non-academic aspects of private schools), it is also possible that parents were not able to easily determine the effectiveness of schools at improving learning outcomes …. (p. 1062, emphasis added)
Two points can be made about this caveat. First, the suggestion of this paper is that we do not know what the results for test scores in mathematics were, given that different tests were used in English- and Telugu-medium schools, so it is not clear if parents are really so poor at judging school effectiveness. However, there are suggestive results, comparing only Telugu-medium schools, which point to improved performance in all subjects when the same tests are used; this could well be the outcome of an experiment in private schools in general which overcame the problem of using different tests.
Second, this caveat also raises a possible conflation of two different meanings of school choice. In the research paper, the ‘School Choice’ of the title means choice through top-down reform, where children are given vouchers to leave public to go to private schools, and it is this major policy debate which is of concern to the authors. However, this is not the only kind of school choice—the other is where parents choose to go to private schools and pay out of their own funds. Elsewhere I have dubbed these two approaches ‘School Choice’ (the capitals signifying a top-down approach, where typically governments fund parents through vouchers etc., to attend private school) and ‘school choice’ (small letters indicating the spontaneous order of parents choosing private schools and paying themselves) (Tooley, 2014).
It is worth making this distinction because in this paper it appears the implications of findings from the first kind of School Choice through top-down reform are illegitimately being transferred across to the second kind of spontaneous choices of parents. The authors suggest that their research findings raise questions about whether the ‘marginal parent’ is justified paying fees to send their child to private school. But parents paying fees for schools was not the object of this research, which was instead based on parents receiving vouchers, a completely different approach. results of an experiment on vouchers (School Choice) won’t necessarily apply to parents paying fees (school choice).
In fact, some results are given in passing for this spontaneous kind of ‘school choice’. For instance the authors report that there are ‘large cross-sectional differences in math and Telugu test scores (of 0.65σ)’ (Muralidharan & Sundararaman, 2015, p. 1039) on the baseline tests in favour of private schools, when students had been in either Grade 1 or pre-school for a full year. But, they argue, as their overall findings show no academic benefit to voucher children being in private schools, this suggests that these large differences in favour of private schools ‘are mostly driven by omitted variables and not by differential effectiveness of public and private schools’ (p. 1039). Given the doubts raised about their headline research findings, it may be that these ‘omitted variables’ are not as important as had been thought.
This kind of conflation can be seen in popular commentaries on the research (there is no suggestion that the current researchers agree with these commentaries). One author notes that ‘contrary to popular perception, private schools are not adding value, as compared to government schools … after adjusting for socio-economic factors’ (Karopady, 2014, p. 51), and asks: ‘If private schools are not adding any value, why then do parents still prefer them?’ (p. 52). The Times of India points to parents’ misguided choices of private schools which ‘ironically, have little to do with outcomes’ (Chowdhury, 2015).
Again, these are conflating the two types of school choice. The results referred to are about children using vouchers, not about children whose parents have paid for private school. Indeed, Karopady himself concurs that, after adjusting for socio-economic factors, the children in private schools by their parents’volition (i.e. those paying school fees) perform ‘significantly better than their government school counterparts in all four subjects’ (Karopady, 2014, p. 51). However, just as for Muralidharan and Sundararaman (2015), for Karopady the results from the voucher experiment bring into question this superior private school performance: ‘The findings seem to indicate that the reasons for better performance of … children … who would have gone to private school in any case … may need to be looked for outside the school’ (Karopady, 2014, p. 52). Again, perhaps not to the extent envisaged.
The recent research from Muralidharan and Sundararaman (2015) adds significantly to the research literature on the impact of low-cost private schools. Although the research implementation was hampered by using different tests in non-language subjects for children in different school types, within the research paper there are findings which suggest that when families are offered vouchers to allow them to access private schools, significantly higher academic achievement compared to those children left in public schools is achieved, at only a fraction of the per capita cost of public education. This suggests that allowing children to access private schools using vouchers could be an important policy reform improving educational opportunities and outcomes for the poor.
Notes
- We follow international convention, as did Muralidharan and Sundararaman (2015), rather than the conventions used in Britain and India, and describe government schools as ‘public’, to be contrasted with private schools.
- The study was conducted before the bifurcation of this state—which now consists of two states, Telangana and one still called Andhra Pradesh.
Disclosure statement
No potential conflict of interest was reported by the author.
Notes on contributor
James Tooley is professor of education policy at Newcastle University. His work on low-cost private schools in developing countries won gold prize in the first International Finance Corporation/Financial Times Private Sector Development Competition. He is cofounder and chairman of Omega Schools, Ghana, patron of the Association of Formidable Educational Development, Nigeria, and chief mentor of the National Independent Schools Alliance (India).
References
Assessment of Performance unit (APu). (1988). Mathematical development; A review of monitoring in mathematics. 1978–1982, Parts 1 and 2. Slough: NFEr.
Bashir, S. (1997). The cost effectiveness of public and private schools: Knowledge gaps, new research methodologies and an application in India. In C. Colclough (Ed.), Marketizing education and health in developing countries: Miracle of mirage? (pp. 124–164). Oxford: Clarendon Press.
Chowdhury, S. r. (2015, February 27). Private schools are not adding value: Study. Times of India.
Day Ashley, L., Mcloughlin, C., Aslam, M., Engel, J., Wales, J., Rawal, S. … rose, P. (2014). The role and impact of private schools in developing countries: A rigorous review of the evidence. London: Department for International Development.
Dixon, P. (2003). The regulation of private schools for low-income families in Andhra Pradesh, India: An Austrian economic approach (unpublished PhD thesis). Newcastle University, UK.
Francis, A. (2014, March 6). Why India’s landmark education law is shutting down schools. BBC News. retrieved from http://www.bbc.com/news/world-asia-india-26333713
Gujarat Government. (2012, February 18). The Gujarat Government Gazette Extraordinary, LIII(54). retrieved from http://gujarat-education.gov.in/education/Portal/News/159_1_MODEL%20 ruLES%2029.2.12.PDF
Karopady, D. D. (2014, December 20). Does school choice help rural children from disadvantaged sections: Evidence from longitudinal research in Andhra Pradesh. Economic and Political Weekly, XLIX(51), 46–52.
Kingdon, G. (1996). The quality and efficiency of private and public education: A case study in urban India. Oxford Bulletin of Economics and Statistics, 58, 57–81.
Mantri, r., & Gupta, H. (2014, January 6). Azim Premji, please listen to Steve Jobs and Bill Gates: Not about technology, but about your educational philanthropy. Livemint. Retrieved from http://www. livemint.com/Opinion/301G41LuZJGgs8qLE3clxN/Azim-Premji-please-listen-to-Steve-Jobs-and- Bill-Gates.html
Math Solutions. (2009). Teaching math to English language learners. In Supporting English language learners in math class, Grades K–2. Math Solutions.
Muralidharan, K., & Sundararaman, v. (2015). The aggregate effect of school choice: Evidence from a two-stage experiment in India. Quarterly Journal of Economics, 130, 1011–1066.
Shah, P. (2015, February 27). Retrieved from http://spontaneousorder.in/private-schools-are-not- adding-value-you-be-the-judge/
Tooley, J. (2009). The beautiful tree: A personal journey into how the world’s poorest people are educating themselves. New Delhi: Penguin.
Tooley, J. (2014). The role of government in education revisited: The theory and practice of vouchers, with pointers to another solution for American education. Social Philosophy and Policy, 31, 204–228.
Tooley, J., Dixon, P., & Gomathi, S. v. (2007). Private schools and the millennium development goal of universal primary education: A census and comparative survey in Hyderabad, India. Oxford Review of Education, 33, 539–560.
Tooley, J., & Longfield, D. (2015). The role and impact of private schools in developing countries: A response to the DFID-Commissioned ‘Rigorous literature review’. London: Pearson.
Tooley, J., & Longfield, D. (2016). Affordability of private schools: Exploration of a conundrum and towards a definition of ‘low-cost’. Oxford Review of Education, 42, 444–459.
