Index for assessing the effectiveness of state institutions

On tools for improving the efficiency of state institutions

Dedicated to the memory of Serhiy Loboyko, a man who initiated positive changes.

Authors

The full-scale invasion has accelerated and exacerbated the development of Ukraine as a state and as a society – a community of people. You and I are participants in these tectonic events.

We can see how many heated discussions and even quarrels are taking place in society around this issue. And we understand how much we need tools to improve the interaction between state institutions and society, tools to increase the efficiency of the state and its institutions. Without these tools, just like without weapons at the front, it will be difficult, even impossible, for us to win the battle for our future. We need them now, immediately.

Fortunately, such tools already exist. Some of them have even been tested in practice and are ready for implementation.

In this article, we will talk about one of them – the Public Institutions Performance Index (hereinafter referred to as the Index). This comprehensive methodology was tested at the end of 2024 and is now ready to be implemented in wide practice.

What opportunities does the Index provide for Ukraine and Ukrainians, how is it built, what results were obtained during its testing – all this is discussed below.

Once again, why is the Index needed?

We can see with our own eyes, and this is confirmed by research, how society’s attitude to state institutions is changing in Ukraine. One of the indicators of these changes is trust. In the documentary research film “Trust in” (released in November 2024), Ukrainian and international experts analysed the process of social transformation in Ukraine, including the trust of Ukrainians in each other and in state institutions. The experts came to the conclusion that we, Ukrainians, have become much more conscious and, accordingly, more scrupulous in our attitude to the state and its institutions. At first glance, it seems that the level of trust has decreased. However, a closer look reveals that it is rather a change in the type of trust rather than the level: Ukrainian society is rapidly maturing and has begun to pay much more attention to the activities of state institutions, and has become more demanding of their effectiveness – the benefits they bring to the country and each individual citizen. Therefore, it is more critical of words and more demanding of real efficiency.

At the same time, assessments of the effectiveness of state institutions are still subjective and scattered. There is no single “scale” for assessing the effectiveness of the state and its institutions, and there is no way to compare the assessments of different state institutions.

As a result, there is no possibility of objective analysis and development of ways and approaches to solving acute problems and, ultimately, to increasing the efficiency of both specific institutions and the state as a whole.

One of the tools to address this urgent problem was the Defence Research and Development Cluster (DDC) project, piloted in autumn 2024 by an association of leading experts and research companies with the support of the International Renaissance Foundation and the operational management of the Centre for Innovation Development.

The idea of the Index is to create a tool for the most objective regular assessment by stakeholders (primarily society) of the effectiveness of specific state institutions in performing their functions in various areas of state governance. In simple terms, it is a “thermometer” that regularly and objectively measures how efficiently a particular state institution is performing from the point of view of society.

The formulated and calculated assessment of the effectiveness of the state (represented by its key institutions) enables society to understand the state of affairs more accurately and formulate its tasks for the state. And for the state and its individual institutions – to have a systematic and digitised feedback from society on their work.

In fact, we are talking about building a unidimensional space in which the effectiveness of all key state institutions and the state as a whole is regularly assessed using the same system and scale.

How is the Index built?

The developers of the model, Andriy Kashpur (ReputationLab Ukraine) and Dmytro Khopta (NGO “Towards the Sun”), based the Index on a three-dimensional matrix that includes: a list of functions that make up the general functionality of the state; an assessment of the importance of each function (in the general context of a specific area of activity in state governance); and an assessment of the degree to which a function belongs to a particular state institution.

Dmytro Savchuk (Info Sapiens), together with the authors of the model, developed a mathematical formula for calculating the Index. This formula is stable, meaning it can be used to assess the effectiveness of various state institutions. It also makes it possible to correctly compare the Indices for different periods: today, tomorrow, and the day after tomorrow. This way, the dynamics of changes can be tracked.

The formula consists of an average assessment of the effectiveness of the state’s performance of a particular function, the average degree to which the function belongs to a particular state institution, the average importance of each function, and the number of functions.

The model also makes it possible to calculate sub-indices – assessments of the effectiveness of state institutions separately for each function.

The purpose of the pilot project was to test the developed model in practice, determine the prospects for its application and areas for development/refinement of the methodology.

For obvious reasons, the topic of Defence of Ukraine was chosen for the pilot project. Within the framework of this topic, a list of 9 state institutions was identified that are directly responsible for or have a significant impact on Ukraine’s defence capability. After consultations with defence experts, a list of 19 key functions was formulated, which make up the general function of “Defence of Ukraine”.

Taking into account the limitations of the pilot project, three stakeholder groups were identified (based on the criterion of their degree of involvement in Ukraine’s defence): military personnel, citizens liable for military service, and non-military citizens of Ukraine.

The project was implemented under the operational management of the Centre for Innovation Development and consisted of a set of quantitative and qualitative studies. Quantitative research was conducted by Info Sapiens, and qualitative research was conducted by Active Group.

The results of qualitative and quantitative research were analysed by a group of experts (independently of each other with the final joint discussion and formulation of conclusions) consisting of: Ihor Zhdanov, Ph.D. (Open Policy Foundation, co-founder of the Razumkov Centre), Artem Zakharchenko, Ph.D. (CAT-UA), Vitaliy Kulyk (Centre for Civil Society Research), Yuriy Tsurko, Mgr.

The objective of the qualitative research (focus groups and in-depth interviews) was to test the approaches and architecture of the Index, identify internal qualitative factors influencing the performance assessment and determine stakeholders’ perceptions of the distribution of functions between state institutions.

Quantitative research provided the calculated parameters needed to calculate the Index. In addition, the level of trust and effectiveness of communication of key state institutions in the defence sector was assessed.

Dmytro Khopta, Head of NGO “Towards the Sun”: “There have been no methods in the world to measure the effectiveness of defence institutions in countries at war. However, society needs such data much more than in peacetime, because the country’s survival directly depends on the effectiveness of defence institutions and its assessment by society. We are very grateful for the opportunity to implement the methodology developed by our group and look forward to its systematic application.”

Results of qualitative research

The qualitative research covered two audiences: civilian citizens of Ukraine (2 focus groups in Kharkiv, Kyiv, Ivano-Frankivsk) and experts more involved in military affairs (in-depth interviews 4 with the military and 5 with civil society activists and volunteers).

It is quite natural that both civilians and most of the experts interviewed consider the General Staff of the Armed Forces of Ukraine and the Ministry of Defence to be the key institutions in the defence sector. The respondents also separately identified the “security forces” (MoD, MIA, SSU). Civilian respondents and experts sometimes mention the President (and/or his Office) in the list of institutions.

The majority of Ukrainian civilians interviewed do not have a clear idea of what the general function of ‘Defence of Ukraine’ consists of. However, they can give an assessment of specific functions, such as “construction of fortifications“, “provision of weaponsto theJFO“, etc. This confirmed the basic hypothesis that the overall assessment of the effectiveness of state institutions should be conducted through the assessment of the effectiveness of specific functions that are more relevant and understandable to respondents.

The general (intuitive) assessment of the effectiveness of state institutions in the defence sector shows rather positive feedback on the work of the General Staff of the Armed Forces and the Ministry of Defence, but in general their work is assessed as rather mediocre.

The work of the Government as a whole and individual ministries in particular is generally rated low (with the exception of the Ministry of Digital Transformation). The Government has “collected” almost all the criticism of the government’s actions in general. There are almost no specific positive assessments of the Government or ministries in the focus groups.

The work of regional military administrations and local self-government bodies is assessed by participants in in-depth interviews in terms of their ability to contribute to defence. Those CMAs known to be supportive of the Defence Forces receive mostly positive feedback. Local authorities are highly praised where they effectively deal with the consequences of Russian shelling. However, the lack of information about the activities of some administrations often leads to negative assessments. Local authorities are also criticised for “stealing humanitarian aid”.

Problems with logistics and lack of material support lead to significant criticism of military administrations and local authorities, reducing the level of trust in them and in the government system as a whole.

In contrast, the NSDC and the General Staff of the Armed Forces receive above-average scores due to their visible active work and contribution to the country’s defence capability. However, a general challenge remains the lack of public awareness of the activities of all institutions, which provides opportunities for improved communication and transparency.

Looking ahead, quantitative research and the calculated indices also confirmed this.

In general, qualitative research has confirmed the validity of the approach to the Index, in particular, the formation of a general assessment of the effectiveness of a state institution through a set of assessments of specific functions – much more understandable to the public than “general functionality”. And the removal of the “power bloc” institution from the “Government” to a separate position.

The high level of relevance/request for tools for objective assessment of state institutions and improvement of feedback between state institutions and society was also confirmed.

Results of quantitative research

Quantitative research was conducted by Info Sapiens using the computer-assisted online questionnaire (CAWI) method on the company’s own online panel. A total of 2031 respondents were interviewed as part of the study. The sample structure corresponds to the structure of the population of Ukraine aged 18-64 according to the State Statistics Service as of 01.01.2022, in terms of gender, age categories, settlement size and macro-region. The margin of error of a similarly sized simple random sample does not exceed 2% in general, and by stakeholder group: for the military – 8.4%, for persons liable for military service – 3.7%, for non-military personnel – 2.5%. Quality control included: controlling the time taken to complete the questionnaire and questions for attentiveness, and rotation of answer options was used where possible.

The quantitative research generally confirmed and detailed the findings of the qualitative research and provided the necessary volume of responses to calculate the Indices.

However, there were some surprises. For example, the Defence Intelligence of Ukraine (DIU) is among the three most important institutions in the defence sector, in addition to the General Staff of the Armed Forces of Ukraine and the Ministry of Defence of Ukraine (Figure 1).

When assessing the effectiveness of state institutions (Fig. 2), respondents recognise the Defence Intelligence of Ukraine, the General Staff of the Armed Forces of Ukraine and the Security Service of Ukraine as the most effective.

The least effective are the Government and local authorities.

The MoD, the NSDC and the Ministry of Internal Affairs receive average scores. At the same time, non-conscripts give higher efficiency scores than military personnel for the MoD, MoI, Government, General Staff and NSDC. The situation is similar when assessing changes in effectiveness over the past year.

Looking at the assessment of the state’s effectiveness in the dynamics (current assessments and subjective assessments of changes in effectiveness over the past year) for individual functions, we see that the audience of non-conscripts gives more positive assessments than the military (Figures 3 and 4).

Almost half of the functions (8 out of 19) are in the sector “performed efficiently and efficiency has improved over the last year”. Another 3 functions are in the yellow sector: they are performed INeffectively, but their efficiency has improved. At the same time, a significant number (8 out of 19) of functions are in the red sector: they are NOT performed effectively and their efficiency has deteriorated. Among these functions are many that respondents rated as important (the weight of the function is presented in circle diagrams and detailed in Figure 5).

The “military” stakeholder group (Fig. 4) is more critical in its assessments. They assess the state’s performance of the vast majority of defence functions as ineffective. However, there is some positive dynamics – the effectiveness of some functions is improving.

Based on the collected data, the Indices for state institutions were calculated. In particular, the General Staff of the Armed Forces of Ukraine, the Ministry of Defence, the Ministry of Internal Affairs, the Security Service of Ukraine and the Government (Fig. 6).

It is obvious that the Index scores for most state institutions are negative – less than 50%. It is also obvious that the military are the most critical of the effectiveness of state institutions, while non-military personnel are more positive.

Among the above indices, the Government of Ukraine (except for the MoD and the MIA) has the worst performance. The Security Service, the General Staff and the Ministry of Internal Affairs have the best scores.

It is worth paying attention to the fact that the MIA received rather low scores in the overall efficiency rating (Figure 2), but according to the results of the Index calculation, it became one of the efficiency leaders. The explanation for this may be that the Index takes into account the degree to which each defence function belongs to a specific state institution. Therefore, it is likely that the overall assessment of the effectiveness of a state institution may be lower than its defence-specific efficiency index, as most state institutions have other areas of responsibility. For example, this study did not distinguish between the MoIA and its individual components, such as the National Guard, the State Border Guard Service, etc. Accordingly, the respondents’ assessments of the defence function should be higher than the overall assessment of the ministry’s effectiveness.

It is also quite natural that the closer a stakeholder group is to the war, the more critical its assessment of the effectiveness of state institutions.

An important aspect of using the Index is tracking the dynamics of its changes. To do this, it is necessary to take measurements at least once every six months, or even more often – once a quarter.

Dmytro Savchuk, Info Sapiens research company: “The benefit of this and similar indices is not in specific values, but in the possibility of comparisons between different objects being evaluated and over time. This index allows for a more detailed assessment of perceived effectiveness than a general question, as it is based on respondents’ perception of the importance and responsibility of various government functions.”

Trust, communications and trust in communications

The project also studied the level of trust and margin for error and, as a separate block, the effectiveness of communications of state institutions. These blocks of research were designed to help assess the potential for positive change.

The rating of trust and margin for error (Fig. 7) shows that society has a high level of trust in the key state institutions in the defence sector. All stakeholders also note the high level of their efforts to correct mistakes. Here again, quite unexpectedly, the Defence Intelligence of Ukraine became the leader, while the General Staff of the Armed Forces of Ukraine took only second place.

The Ministry of Internal Affairs has a critically low level of trust. It is worth noting here that the focus groups showed that respondents quite clearly distinguish between the Ministry of Internal Affairs and the National Guard, which is involved in hostilities and is viewed positively. And this was confirmed by the calculated indices, where respondents assessed the effectiveness of the MIA in performing defence functions rather positively (compared to other institutions).

The Government of Ukraine, on the other hand, has a low level of trust and one of the lowest Index scores. Moreover, the public gives it an extremely low rating for its effectiveness in correcting its mistakes. In fact, the rating of trust and the right to make mistakes coincides with the rating of overall effectiveness (Fig. 2) and the Indices (Fig. 6).

Despite the low scores of the Ministry of Internal Affairs, the Government and local authorities, the project’s analytical team assesses the overall picture as indicating that there is a sufficient basis and potential for implementing reforms that will increase the effectiveness of state institutions. And the leading institutions have a sufficient “safety margin” for this. This is evidenced, in particular, by the “margin for error” of the Defence Intelligence of Ukraine, the General Staff of the Armed Forces and even the SSU.

One of the critical components of such reforms is the system of communication with the public. Along with the growing public demand for more effective government, there is also a growing demand for more honest and open communication from the state.

One of the important results of this project was the confirmation of the hypothesis that different groups in our society are highly connected. In particular, this is manifested in the fact that the second and fourth most important communication channels are receiving information “personally from the military” and “friends and acquaintances” respectively. With the corresponding highest level of trust in the military and volunteers as speakers and an extremely low level of trust in official speakers (Fig. 8).

These and other factors have obviously influenced the stakeholders’ assessment of the effectiveness of communications of state institutions in general (Fig. 9).

Comparing the Indices (Fig. 6) and the communication effectiveness rating (Fig. 9), we see that public communications influence the assessment of the effectiveness of public institutions, but it is the only and dominant factor of influence. As noted above, Ukrainian society is communicatively connected – Ukrainians largely exchange information (assessments and judgements) directly outside official channels, whose speakers are also extremely trusted. In other words, Ukrainians “know how it is in reality”, not in the media, and actively exchange information among themselves.

The study found that most state institutions failed in their communication with society. This is also because they failed to convince the public that their information and their speakers are trustworthy.

This creates an additional challenge for state institutions: how to restore trust in themselves if your communication with the public is extremely ineffective. After all, it is not enough to develop and implement positive changes and increase your efficiency. Now you also need to communicate them in a way that makes the public believe you.

The project’s analytical team believes that the key to solving this problem is the introduction of dialogue communication formats that should replace “pr-monologues”. This is actually one of the objectives of the Index project as a whole.

Conclusion

Although the Defence Institutions Performance Index project is only a pilot, it has demonstrated a rather sad but natural current situation: Ukrainian society does not satisfactorily assess the effectiveness of most state institutions in the defence sector.

At the same time, there are significant positive aspects.

Firstly, society is clearly becoming more conscious and scrupulous in its attitude to the state and state institutions. Accordingly, the relevance of tools for objective assessment of the effectiveness of the state as a whole and its individual institutions is growing dramatically.

Second, a significant number of functions have a positive assessment of their performance, as do key state institutions in the defence sector. Accordingly, projects such as the Index allow for a more accurate and timely determination of which specific state actions or reforms of which functions were most positively assessed by society.

Thirdly, key state institutions in the defence sector, primarily the General Staff of the Armed Forces of Ukraine, have significant potential (including public trust) to carry out reforms and other actions that will increase their effectiveness. And the introduction of the Index as an objective assessment system reduces the field for enemy manipulation and fostering a sense of total betrayal.

The main conclusion of the project is that there is a unique situation now, when the public demand for increasing the effectiveness of the state is growing rapidly, and at the same time there is an opportunity to create and implement objective assessment tools that will reduce the field for manipulation and provide valuable information for real improvement of the work of state institutions and the state as a whole.