Methodology

The state of political transformation (Democracy Status), is measured in terms of five criteria, which in turn are derived from assessments made in response to 18 individual questions. The BTI’s conception of democracy goes well beyond other definitions of democracy, which are limited primarily to basic civil rights and the conduct of free elections. Stateness, which is seen as a precondition to political transformation, is included in the BTI’s definition of democracy and examined through questions specifically dealing with the state’s monopoly on the use of force and basic administrative structures. It also includes an evaluation of the rule of law with an eye to the separation of powers or the prosecution of office abuse. In addition, it assesses the degree to which the democratic system is consolidated in terms of its acceptance, representativeness and political culture.

The state of economic transformation (Market Economy Status), is comprised of seven criteria, which are based on a total of 14 individual questions. The BTI’s concept of a market economy includes not only aspects such as economic performance, regulatory or competition policy and property rights; it includes elements of social justice such as social safety nets, equality of opportunity and sustainability as well. In BTI terms, comprehensive development not only aims at economic growth, but also requires successful poverty alleviation and the freedom of action and choice for as many citizens as possible.

The Management Index, which focuses on how effectively policymakers facilitate and steer development and transformation processes, is the key innovation of the BTI. By examining and evaluating decision makers’ reform policies, the BTI sheds light on those factors determining success and failure on the way to democracy and a market economy. Successful transformation management implies that governments are consistent in pursuing their goals, and use their resources wisely and effectively. It also implies that decision makers win the broadest possible consensus for their transformation goals, and work reliably with external supporters and neighboring states. With its focus on political actors’ steering capacity, the BTI is the only index to analyze and compare governance performance with self-collected data.

The level of difficulty, which is constructed from three qualitative and three quantitative indicators, reflects the observation that each country’s quality of transformation is influenced by structural conditions. In this way, difficult initial conditions and scarce resources in a country are factored into the equation for political management performance.

Because the BTI focuses on the normative goals of democracy under the rule of law and a market economy anchored in principles of social justice in its analysis of political decision makers targeting these goals, it excludes from its survey countries which might be considered long-consolidated democratic systems and in which economic development can be regarded as well-advanced. This is not to suggest that with the achievement of these two goals, a static end-state is achieved. Rather, it reflects the observation that reform needs and priorities following the consolidation of democracy and a market economy will differ from those that emerge during transformation processes.

In the absence of a clearly defined “threshold of consolidation”, the Transformation Index excludes all countries that were members of the Organisation for Economic Co-operation and Development (OECD) by the year 1989. An assessment of political performance, governance and the quality of democracy in OECD member states is offered by the Sustainable Governance Indicators (SGI), a project also conducted by the Bertelsmann Stiftung (www.sgi-network.org). Means of extending the future selection of countries beyond this makeshift criterion of formal OECD membership are currently being reviewed.

Due to limited project resources, small states with fewer than 2 million residents (until the BTI 2006: 3 million) were not considered in the BTI. However, because the strict application of this criterion would have led to the exclusion of some particularly interesting examples of development and transformation, Bahrain, Bhutan, Estonia, Kosovo, Mauritius, Montenegro and Qatar are also included as exceptions.

Since 2003, the number of countries surveyed has increased from 116 to 128. They are divided into seven regional groups: South and East Africa (19 countries), West and Central Africa (18), Asia and Oceania (21), Middle East and North Africa (19), Latin America and the Caribbean (21), Post-Soviet Eurasia (13), and East-Central and Southeast Europe (17).

The Transformation Index is based on a qualitative expert survey, in which written assessments are translated into numerical ratings, and examined in a multi-stage review process so as to make them comparable both within and across regions. This method enables those factors of political and economic development that elude purely quantitative assessment to be captured in the experts’ qualitative appraisals, a significant advantage. For example, a distinction can thus be made between rights granted de jure and their de facto implementation. In addition, statements can be made about the magnitude of social capital and the extent to which civil society is integrated into political decisionmaking processes. Furthermore, governance quality can be assessed and compared. Given facts, such as constitutional provisions or official economic data, can be interpreted and weighed in context. The resulting country assessments render fully transparent and verifiable the grounds for each of the BTI’s 6,656 individual scores.

Nevertheless, this type of qualitative expert survey still retains a degree of subjectivity. The BTI survey process takes this into account during the preparation of reports and evaluations, as well as during the review of the data. It is designed to minimize subjective factors as far as possible throughout the process.

The process of country assessment has both a qualitative and quantitative component, in each case performed by two country experts. As a rule, one foreign and one local expert are involved in the evaluation process; this ensures that both external and internal perspectives are taken into account in the course of assessment, and increases the level of objectivity. In total, 236 experts from leading research institutions around the world contributed to the production of the country reports and the assessment of the individual questions.

A standardized codebook serves as the foundation of the survey process, providing a single reference framework for the qualitative and quantitative answers to the questions, and explaining the various criteria and questions. The first expert drafts a detailed report on the basis of the criteria outlined in the codebook, referencing the 49 total questions associated with each criterion. The second expert reviews, comments and adds to this country report. In addition, in the course of answering 11 of the 49 individual questions, the country experts are required to draw upon a set of 36 external quantitative indicators (ranging from inflation rates to education spending). Independently of one another, the two country experts translate the assessment into a numerical rating on a scale of one (the lowest value) to 10 (highest value), structured by four levels of score-based categories contained in the codebook. In this way, countries are evaluated on the basis of whether and to what extent they comply with the specified rating levels and fulfill the BTI criteria.

In order to ensure the validity, reliability and comparability of the assessment, each individual score undergoes a multistep process of review by the country experts, the regional coordinators, the project team and the BTI board. The numerical expert ratings and the written explication of each of 49 individual questions are initially reviewed by regional coordinators, who examine the content to ensure it is both complete and consistent. The regional coordinators, all political scientists with an expertise in comparative studies, participate in each step of the report-creation process, and apply their regional expertise to ensure the high quality of the country reports. They subsequently perform an intraregional calibration of their countries’ scores, before they join with the project team to carry out an interregional score calibration for all 128 countries, this time checking for across-the-board comparability and viability. Finally, all scores are discussed once again by the BTI board before being adopted. The BTI board, a panel of respected scholars and experienced practitioners in the field of development and transformation, provides the project with ongoing support and advice.

The Status Index is formed by calculating the average of the total scores given for the dimensions of political (Democracy Status) and economic (Market Economy Status) trans- formation. The state of transformation in each analytic dimension is equivalent to the average of the scores of the associated criteria. Criterion scores are in turn based on the average scores of the equally weighted individual questions that comprise the criterion.

Combining the two analytical dimensions into a Status Index follows the normative premise of the BTI, under which transformation is always conceived as a comprehensive transition toward democracy and a market-economic system. Doing so underscores the close empirical and functional relationship between democracy under the rule of law and a socially just market economy, though this should not imply any automatic, linear development process or fixed sequence of transformation steps and causalities. On the other hand, the high level of aggregation in the Status Index makes it more difficult to quickly identify whether political shortcomings or economic deficiencies represent significant obstacles to transformation. Therefore, in contrast to previous editions, the BTI 2012 presents the state of political and economic transformation as separate rankings, in order to enable users to easily locate countries’ varying levels of development in terms of democracy and a market economy.

The Management Index is formed by calculating the average of scores given for the management criteria, which is then offset against the assigned level of difficulty.

The individual questions on the state of political transformation are also used by the BTI in determining whether a country is classified as a democracy or autocracy. This analysis comprises more than just whether sufficiently free and fair elections are held.

In accordance with the Transformation Index’s comprehensive conception of democracy, six threshold values are initially considered. The country is classified as an autocracy if even one score falls short of the appropriate threshold.

The group of autocracies also includes “failed states”, defined as countries in which the state’s monopoly on the use of force and basic administrative structures are either lacking overall or encompass only a part of the territory or population, so that the government is severely limited in its capacity to act (the average of scores given for questions 1.1 and 1.4 is less than three).