Cracking the code: understanding successful civil service reform

My civilservicereformuk.com research programme is looking differently at civil service reforms to provide insights into the practice of reform that will help ministers and officials succeed in future reforms.

For the last two years I have been searching for perspectives that could shed fresh light on why and how reforms work; what practitioners actually do; and above all how it was that the reform became embedded in changed ways of thinking and changed ways of working.

The result is my conceptual framework for understanding civil service reform.

This blog outlines the research that has shaped this framework.

In my mission impossible blog last year I made the case for taking a different view of the impact of successful reform. It seems inevitably fruitless to try to attribute causality for improvements to civil service effectiveness to elements of specific reforms. Even if there was any kind of credible non-tautological measure of civil service effectiveness (spoiler alert – there isn’t) each link in any causal chain is contested: buffeted by context, culture, power, stakeholders and a complex array of variables. Reforms do not come as neat interventions. They are a messy agglomeration over decades, sometimes taking 20 or 30 years from the simmering of the primordial soup of reform ideas to becoming embedded within the civil service. The path dependency of reforms means causality cannot be attributed to one phase of reform. Later reforms only became possible because of previous reforms. A more productive approach may be to look at how reforms build dynamic capabilities which in turn enable innovation, effectiveness, improvement and continuous adaptation.

I described the limited success of public administration research in answering questions about the what and how of reform success in a parade of paradigms. An increasing number of researchers have sought to address the theoretical limitations of the public administration tradition by drawing on theories from other fields. They have used theories from strategy process, strategy practice, institutional work, sensemaking and dynamic capabilities  - often in combination  - to understand change in public sector institutions.

I found six research fields (which are increasingly interlinked) best placed to expose the reality of successful civil service reform:

·      Strategy process;

·      Strategy as practice;

·      Institutional work;

·      Strategy discourse and sensemaking;

·      Dynamic Capabilities; and,

·      the Multiple streams approach.

I have produced a short summary of each of these research fields in how research can help on my website, but in this blog I share what I took from them to produce the first version of the conceptual framework that will guide my further research.

My conceptual framework

In order to draw out key themes from my research I coded my notes from the 50 most compelling articles and books. I began with codes reflecting the 7 questions that I used to guide my literature review, and then adapted the coding structure to reflect the issues that emerged from my notes. I then developed those themes into a draft conceptual framework for an episode of civil service reform.

Figure 1. A conceptual framework for an episode of civil service reform

Source: Peter Thomas

Catching reality in flight

Research into how organisations and institutions change has evolved tremendously in the 40 years years since I studied management science (still an uncommon degree amongst civil servants) at Warwick Business School.

Over time there has been a welcome shift away from theories of change dominated by public choice theory to approaches that are instead rooted in sociology. This change has led to more plausible frameworks for looking at the emergence, design and implementation of reforms in the civil service.

The father of the now dominant research fields of ‘strategy process’ and its offspring ‘strategy as practice’ (SAP) carried out a landmark study of change in the regions of the NHS in the early 90’s (A. M. Pettigrew et al., 1992). His starting point was a critique of most existing research on organisational change as ‘ahistorical, acontextual and aprocessual’.  Instead he proposed a view of strategy in which strategy content is the output of a ‘legitimisation’ process which although expressed in rational terms is shaped by political and cultural considerations  - ‘politics as the management of meaning’ (A. M. Pettigrew, 1987).

His ambition was to ‘catch reality in flight’ within the context of ‘the ongoing processes of continuity and change’ in order to displace the then dominant rational theories of choice and planned change. (A. Pettigrew, 2013).

‘Actions and actors drive processes but actions are embedded in multiple levels of context and both the actors and the context are shaped and are shaping - the interchange between agents and contexts over time is cumulative - the legacy of the past is always shaping the emergent future.’ (Pettigrew, 2012)

In the 20 years that followed Pettigrew’s landmark article, process research scholars attention has gone beyond decision making, change and the managerial elites to encompass amongst other things: the role of middle managers and value of involving other employees – the notion of open strategy; exploring the importance of framing and cognition and how the attention of the organisation can be affected through discourse – and consequently seeing the process of strategy as having some features of a social learning process; and, seeing the value of a parallel stream of resource thinking on capabilities, especially the growing work on micro foundations of dynamic capabilities as outcomes of the strategy process, whilst the process itself can be seen as a dynamic managerial capability in its own right (Burgelman et al., 2018).

Researchers are increasingly looking at where closer collaboration between could be fruitful, including institutional theory, strategy process and dynamic capability. (Jarzabkowski et al., 2022)

Context is king: reform is a fragile plant

Public administration research has repeatedly highlighted how context and antecedents play a key role in enabling or hampering reform efforts.

Pettigrew distinguishes outer context (national economic, political and social context, social movements and long term professionalisation) – from inner context (the ongoing strategy, structure, culture, management and political process of the organisation). The process of  change encompasses the actions, reactions and interactions of the various interested parties as they negotiate around proposals for change. He sees the role of actors in change in mobilising the contexts around them to provide legitimacy for changes as a critical connection that is made between context, content and process in pursuit of change (A. M. Pettigrew et al., 1992).

A large review of 519 studies of NPM impact across Europe (Pollitt & Dan, 2013) concluded that whilst NPM interventions could not be called a failure, the political, structural and cultural context was crucial to the success of NPM interventions. They compared NPM interventions to ‘a delicate plant [that] requires the right soil and care, more orchid than potato’. As well as being intrinsically hard to evaluate, the importance of the context to each intervention complicates the attribution of the causes of any outputs and impacts (Pollitt & Dan, 2013).

The practice turn: how people do successful change

In reflecting on the evolution of strategy process research 20 years after his landmark articles Pettigrew repeated his earlier call for further research into the how of change, focusing particularly on the intentions and actions of key agents (A. M. Pettigrew, 2012). The emergence of Strategy as practice (SAP) is seen by some as a response to that call (Sminia & de Rond, 2012).

SAP emerged in the early 2000’s with the promise of being able to provide insights into the tools and methods of strategy-making (practices), how strategy work takes place (praxis), and the role and identity of the actors involved (practitioners) (Vaara & Whittington, 2012).

With its roots in sociological theory, SAP examines how agents ‘are enabled by organizational and wider social practices in their decisions and actions’ whilst drawing on the strategy process approach. It has links with other theories found in strategic management including ‘sensemaking’ and the dynamic capabilities perspective (Vaara & Whittington, 2012).

SAP also aims to relate individual level micro-activities to both the organisational level, and wider multi-organisational field. It sought to remedy the lack of attention to the role of people and agency in the mainstream strategy research of the time. (Jarzabkowski et al., 2022).

The three p’s framework – practitioners, practices and praxis have remained at the heart of the SAP framework but as the field has matured researchers have drawn on a wide range of theoretical and methodological approaches, some new to the strategy field, including: structuration theory, sensemaking and discourse. (Jarzabkowski et al., 2022)

Why practitioners do what they do

SAP has been criticised for conceptualising strategy as little more than a kind of empirical objective ie: the doing of strategy (Rouleau and Cloutier 2022). Some argue this has led to an excessive focus on capturing the activities and micro level actions of managers at the expense of shedding light on why they are doing what they are doing, most importantly failing to seek a substantive impact on organisational outcomes (Suddaby et al., 2013). Rouleau and Cloutier suggest that researchers have neglected the nature of social practice, which should include the collective knowledge that agents have acquired over time and their contingent use of this in their social and organisational context (Rouleau & Cloutier, 2022). This argument reinforces the relevance of wider professional fields outside the civil service in understanding what shapes and drives the practice of reform.

The heavy hand of history: path dependency

The importance of path dependency is clear from public administration research. One of the architects (Kate Jenkins) of the most impactful UK reform – Next Steps Agencies is clear that they drew on what went before, and created the platform for what happened next:

I do not say that Next Steps is a tremendous success because there are 103 agencies 10 or 15 years later. I say that it is a great success, as the FMI was a great success, because it has led on to the next thing, which is relevant to how the Civil Service is operating now. That is the real story of Civil Service reform. (Kandiah, M., 2007).

Few reforms can be understood in isolation from what went before, or in ignorance of the contemporary political, economic and social context. To transplant a reform structure (such as a delivery unit) from one setting to another without understanding the path dependency and context of the original reform is to guarantee the rejection of that reform in its new host.

Change is not solely determined or controlled by senior leaders.

The approach of public administration scholars to change and role of key actors has been criticised for taking a too narrow view of change as story of top down versus bottom up forces, and trying to explain success and failure without exploring the role of middle managers in departments and agencies who have to perform the change if it is to succeed (Cloutier et al., 2016). These middle managers play a critical linking or mediating role between top down strategic intent and local operations (A. M. Pettigrew et al., 1992)(Mantere, 2007).

More recently the concept of ‘open strategy’ (Jarzabkowski et al., 2022) has fuelled the emergent view that strategy and change is and should be made by many people at different levels in an organisation. And because change is not a walled garden for the senior management team or strategists we need to talk to a wider group than the usual cast list of permanent secretaries, ministers and director generals if we are to understand the course of successful reforms.

Agents, coupling and alignment: how reform ideas come into play

Kingdon’s (2014) multiple streams approach (MSA) published in 1995 provides a compelling picture of the messy reality of how ‘an idea’s time comes’ in government. It exposes the fluid and informal nature of a number of agents who play a key part at different points in the policy process. Whilst its origins lie in sociological theory, its application has overwhelmingly been in political science.

He describes three streams (problems, policy, politics) each with various sub-components. These streams co-exist independently until the point where a policy window opens to create an opportunity for a few “policy entrepreneurs” (I will subsequently use the term “agents”) to push their conception of the problem and the solutions.

He describes the function which these agents serve in the system as one of ‘coupling’ the three streams to the extent that they are sufficiently aligned to substantially increase the prospect of their ideas being adopted on an agenda for decision. They are playing the role of broker and bricoleur as well as advocate.

The influence and appeal of his multiple streams framework is reflected in a large body of work which seeks to apply and extend it through all the stages of the policy process, and to combine it with other concepts.

There have been some notable efforts to develop Kingdon’s approach to address some of the main criticisms of his work.

Ackrill and Kay (2011) make several necessary changes to Kingdon’s conceptualisation of policy entrepreneurs (or agents):

• Policy makers in government are not just passive agents being sold solutions by policy entrepreneurs, but in fact acting as entrepreneurs themselves in making choices about which agents and solutions are suitable to the window and should become part of the coupling of streams.

• “Policy entrepreneur” is better used as a label for a set of behaviours at moments in a policy process, not as a permanent characteristic of an agent in the process. They found that agents can help create the windows of opportunity, rather than passively waiting for one to open.

• Agents are both inside and outside the system and hierarchy and may move between the two. They take on different roles through the process.

Howlett et al (2015) extended the original stream framework beyond the agenda setting phase to policy development and decision making. They add a process stream and a programme stream to explain how at each subsequent confluence point different actors, powerful shifts in ideology and interest can change or reframe problem definitions in an existing policy flow. They claim that their model enables the factoring in of the role of various agents as active steerers of the policy ‘ship’, or alternatively explains how actors try to reconfigure or even block the flow of the policy. My unscientific personal and opportunistic sample of reformers and policy makers revealed that these MSA metaphors and models resonate with practitioners in way that most of the paradigms of public administration did not.

Embedding reform: engagement and fit

Boswell and Rodrigues (Boswell & Rodrigues, 2016) argue that MSA has demonstrated that it can be used to analyse how policies are applied and implemented across sectors or levels of government. They proposed that‘how this policy is implemented then depends on its confluence with local or sectoral problem and politics streams.’ They predict that where external requirements (for example from the centre of government) do not fit with departments’ beliefs and conception of their local organisational problems, implementation will be weaker.

They tested their explanatory typology of implementation modes on a few UK reforms (including one, Asylum Targets, for which I was the lead official in the Prime Minister’s Delivery Unit at the time). In this case they observed that modes of implementation shifted over time (from coercive implementation to bottom-up implementation) as the organisation came to see the solution as fitting with their local framing of organisational problems. It is this fit – even as the politics stream behind the original reform moved on – that explains why the organisation embedded the solution and applied it to other problems.

Institutional Work scholars have found that the ambiguity and flexibility provided by a broad reform vision rather than a blueprint allows middle managers to adapt it to local circumstances and capacity and therefore improve the prospects for successful implementation (Cloutier et al., 2016). These insights lead to different questions about the role of senior leaders and core reform teams: how do they enable and support this role of middle managers in the way they frame the reform and conduct the process of design and implementation (Mantere, 2007).

As the stage of reform moves onto engaging, mobilising, designing and implementing action there are some increasingly convergent themes in both strategy research fields and the field of institutional work. As reforms are designed and enacted the logic of change is one of learning through experience, discourse and sensemaking (Balogun et al., 2014).

Outputs and impact

In mission impossible I made the case for exploring the potential of using some dynamic capabilities as a better way to capture the impact of successful reforms. My hypothesis is that the most successful reforms became embedded in changed ways of thinking and changed ways of working. These new routines formed dynamic capabilities that enable (but don’t guarantee) a more effective and innovative civil service.

Several strategy researchers have made the same connection, ‘the output of good change process is heuristics: high performing processes which create dynamic capabilities linked to performance’ (Bingham et al., 2007).

In the 1990’s dynamic capabilities (DC) were introduced as a framework to explain the ability of an organisation facing a rapidly changing environment to know their context and reconfigure their assets to maintain their competitiveness. These DCs were described as being organisational learning skills: generating new knowledge that would ‘reside in new pa7erns of activity or routines’ (Teece et al., 1997). DCs have become the dominant theoretical framework for understanding how organisations change (Piening, 2013)

Teece later broke DC’s down into three kinds of capacities: ‘(1) to sense and shape opportunities and threats, (2) to seize opportunities, and (3) to maintain competitiveness through enhancing, combining, protecting, and when necessary, reconfiguring the business enterprise’s intangible and tangible assets’ (Teece, 2009). DCs acted as strategic meta-routines through which organisations adapt, change or introduce new operational routines to improve their performance.

A seminal article (Zheng et al., 2011) argued that ‘the fundamental function of the firm is to integrate and use knowledge’ and drew out three Knowledge Building Capabilities that stand above others as special kinds of DCs: knowledge acquisition capabilities, knowledge generation capabilities and knowledge combination activities. They found that this final knowledge combination capability contributed the most to innovation and performance, drawing on the raw material created by the other two processes. They noted the increasing role that alliances and networks play in organisations’ environment.

Research has increasingly focused on exploring the range of processes and routines that provide the micro-foundations as the building blocks for DCs, explaining how they are created (Bhardwaj et al., 2022). More recent researchers typically identify micro-foundations of specific DCs as ‘second order’ themes which are underpinned by ‘first order’ routines (Ince & Hahn, 2020) (Bhardwaj et al., 2022). Different firms can reach similar DCs by quite different routes reflecting their own context and history. A number of antecedents, for example the degree of the organisation’s embeddedness in networks and alliances, are proposed as important aspects of the path dependency of dynamic capabilities (Zheng et al., 2011).

DCs seem to provide a potential lens for viewing the success of civil service management reforms. Whilst the approach developed overwhelmingly through research in private sector organisations to those in the public sector, and specifically the civil service. An emerging body of work seems to make a plausible case (Pablo et al., 2007), (Piening, 2013), (Ince & Hahn, 2020), (Bhardwaj et al., 2022).

The two interrelated frameworks of (Teece, 2009) and (Zheng et al., 2011) have both been applied in non-private sector settings. (Feldman & Pentland, 2003), (Piening, 2013), (Ince & Hahn, 2020), (Vallaster et al., 2021), (Wenzel et al., 2021). These studies helpfully identify common DCs and micro-foundations with private sector studies but also exposed some distinctive DCs and micro-foundations that reflected the tensions facing non-profits and hybrid organisations (Bhardwaj et al., 2022) (Vallaster et al., 2021).

I first encountered the notion of routines and dynamic capabilities back in 2007, when as the leader of the capability review programme I commissioned the Sunningdale Institute (SI) to carry out an evaluation of the programme – which was the UK Cabinet Secretary’s (Lord O’Donnell) flagship reform. The Sunningdale Institute was a virtual academy of thought leaders from the UK and elsewhere, primarily in management, organisation and governance – sadly and shortsightedly wound up at the creation of the Institute for Government. One of their recommendations was that Civil Service leaders needed to plot a path towards a system characterised by the merits of dynamic capability. I was especially struck by how they suggested the civil service might start on that path

‘It is unrealistic to expect rapid, system-wide development of dynamic capabilities at this point. Instead, we recommend picking a small number of areas – perhaps six areas including a combination of some units within departments and one or two cross-cutting issues – to act as trailblazers or beacons for the wider Civil Service.

It was another 10 years or so before I saw others bring this approach to thinking about dynamic capabilities in UK public services – the excellent Institute for Innovation and Public Purpose IIPP) based in University College London (UCL) (Kattel & Mazzucato, 2018).

The IIPP’s work shows the value of looking at reform success through the lens of dynamic capabilities. One of the trio of international case studies in Kattel’s excellent research (2022) was the Government Digital Service (GDS) in the UK. He proposed three set of routines as the key elements of dynamic capabilities in public sector organisations:

• Sense-making routines: analytical, assessment, information-gathering and processing routines that enable new learning, appraisal and evaluation patterns. These routines can relate to analysing outputs and outcomes (value), as well as the internal performance of an organisation.

• Connecting routines: networking and boundary-spanning routines that enable new networks and coalitions of internal and/or external stakeholders to be built. The routines help to (re-build legitimacy and buy-in for new solutions.

• Shaping routines: routines to design and implement specific new directionality for an organisation or policy area, embed and mainstream new solutions into long-term routines, either in policy or in management, and be able to provide resources and support for new initiatives.

It is significant for those interested in the conduct of successful reforms that he found three crucial sources of dynamic capabilities in his case studies (Kattel, 2022):

1. Political leadership was critical to initiating or supporting the creation of dynamic capabilities.

   
   

2. New managerial leadership. In every case the role of new managers was critical in the introduction and embedding of dynamic capabilities. Intriguingly none of the new managers had worked in the civil service, although they had been involved in public service projects.

   
   

3. The creation of a new organisation. The GDS stood out in Kattel’s research, showcasing how ‘the autonomy of a new organisation, capable new leadership and high-level political support created conditions for the new organisation to be highly effective in actually implementing changes’.

This stream of work provides strong encouragement for me that there will be substantial value from developing further reform case studies with the sources and routines of dynamic capability in mind. It is encouraging that the test and learn programme in the cabinet office’s public services reform team in 2025 displays similar characteristics.

What next?

I am developing a series of reform case studies which will use the lens of this draft conceptual framework. These cases will also explore the feasibility of using dynamic capabilities as an indicator of reform impact and success.

Later, in 2026 I will use that tested and revised framework to guide pulses of primary research with past and present reform practitioners.

Peter Thomas 12^th July 2025

References

Ackrill, R., & Kay, A. (2011). Multiple streams in EU policy-making: The case of the 2005 sugar reform. Journal of European Public Policy, 18(1), 72–89. https://doi.org/10.1080/13501763.2011.520879

Balogun, J., Jacobs, C., Jarzabkowski, P., Mantere, S., & Vaara, E. (2014). Placing Strategy Discourse in Context: Sociomateriality, Sensemaking, and Power: Placing Strategy Discourse in Context. Journal of Management Studies, 51(2), 175–201. https://doi.org/10.1111/joms.12059

Bingham, C. B., Eisenhardt, K. M., & Furr, N. R. (2007). What makes a process a capability? Heuristics, strategy, and effective capture of opportunities. Strategic Entrepreneurship Journal, 1(1–2), 27–47. https://doi.org/10.1002/sej.1

Boswell, C., & Rodrigues, E. (2016). Policies, politics and organisational problems: Multiple streams and the implementation of targets in UK government. Policy & Politics, 44(4), 507–524. https://doi.org/10.1332/030557315X14477577990650

Burgelman, R. A., Floyd, S. W., Laamanen, T., Mantere, S., Vaara, E., & Whittington, R. (2018). Strategy processes and practices: Dialogues and intersections. Strategic Management Journal, 39(3), 531–558. https://doi.org/10.1002/smj.2741

Cloutier, C., Denis, J.-L., Langley, A., & Lamothe, L. (2016). Agency at the Managerial Interface: Public Sector Reform as Institutional Work. Journal of Public Administration Research and Theory, 26(2), 259–276. https://doi.org/10.1093/jopart/muv009

Howlett, M., McConnell, A., & Perl, A. (2015). Streams and stages: Reconciling Kingdon and policy process theory: Streams and stages: Reconciling Kingdon and policy process theory. European Journal of Political Research, 54(3), 419–434. https://doi.org/10.1111/1475-6765.12064

Kandiah, M., L., R. (2007). The Civil Service Reforms of the 1980’s. CCBH Oral History Programme.

Kattel, R. (2022, July). dynamic capabiliies of he pulic sector: Towards a new synthesis. UCL.

Kattel, R., & Mazzucato, M. (2018). Mission-oriented innovation policy and dynamic capabilities in the public sector. Industrial and Corporate Change, 27(5), 787–801. https://doi.org/10.1093/icc/dty032

Kingdon, J. W. (2014). Agendas, alternatives, and public policies (2 ed., Pearson new international ed., update ed. with an epilogue on health care). Pearson.

Mantere, S. (2007). Role Expectations and Middle Manager Strategic Agency. Journal of Management Studies, 0(0), 071106213007003-??? https://doi.org/10.1111/j.1467-6486.2007.00744.x

Pettigrew, A. (2013). The Awakening Giant (Routledge Revivals) (0 ed.). Routledge. https://doi.org/10.4324/9780203816264

Pettigrew, A. M. (1987). CONTEXT AND ACTION IN THE TRANSFORMATION OF THE FIRM. Journal of Management Studies, 24(6), 649–670. https://doi.org/10.1111/j.1467-6486.1987.tb00467.x

Pettigrew, A. M. (2012). Context and Action in the Transformation of the Firm: A Reprise: Context and Action in the Transformation of the Firm: A Reprise. Journal of Management Studies, 49(7), 1304–1328. https://doi.org/10.1111/j.1467-6486.2012.01054.x

Pettigrew, A. M., Ferlie, E., & McKee, L. (1992). Shaping strategic change: Making change in large organizations: the case of the National Health Service. Sage Publications.

Rouleau, L., & Cloutier, C. (2022). It’s strategy. But is it practice? Desperately seeking social practice in strategy-as-practice research. Strategic Organization, 20(4), 722–733. https://doi.org/10.1177/14761270221118334

Suddaby, R., Seidl, D., & Lê, J. K. (2013). Strategy-as-practice meets neo-institutional theory. Strategic Organization, 11(3), 329–344. https://doi.org/10.1177/1476127013497618

Vaara, E., & Whittington, R. (2012). Strategy-as-Practice: Taking Social Practices Seriously. Academy of Management Annals, 6(1), 285–336. https://doi.org/10.5465/19416520.2012.672039