Dr Wayne Smith, Head of Economics, the Health Economics Unit, discusses how evaluations can assess the impact of innovations and technological developments in health care and support their adoption into the NHS.

As healthcare innovators develop novel medical devices and digital healthcare solutions to add to the medicines arm of health care, the evaluation of such innovation needs to keep pace with the rate of technological development.

Innovation in healthcare can take several forms, including new models of care, digital technology, invitro diagnostic tests and other medical devices. To facilitate their introduction into mainstream patient care, it is important that the right evaluation approaches are used for each type of innovation, taking into account any gaps in the evidence. In this blog, I’ll discuss:

• The purpose and perspective of an evaluation
• Selecting or developing an appropriate framework for an evaluation
• The maturity of the technology and identifying the gaps in the evidence already generated for evaluation purposes
• Types of evaluations, ‘Formative’ and ‘Summative’
• The HEU approach. 

The purpose and perspective of an evaluation

Evaluations are performed to determine the short, medium to long-term consequences (impact) of applying a technology and to inform decision making and. This includes looking at whether the technology is safe, whether it worked in a research setting (efficacy), whether it worked in a real-world setting (effectiveness), and its value for money (efficiency).

For example, an evaluation of a digital solution such as a remote monitoring system (RMS) for diabetes will examine the interaction of users and the health services with the digital solution and then measure the changes attributable to the intervention. This helps to answer the question What changes in behaviour, processes or health outcomes can be attributed to the intervention?

Evaluators are usually asked to examine a new technology from the NHS perspective. Where budget holders need to make decisions on adopting technology, their decisions are largely influenced by the benefits to patients, value for money – cost effectiveness of an intervention compared to

Selecting or developing an appropriate framework for an evaluation

A conceptual framework, for example a logic model, can be used to outline the objectives of a programme or intervention and to demonstrate the expected causal relationship between inputs, processes, outputs and impacts. This is rooted in cause and effect and can take a simple diagrammatic form. It can be used as the backbone for developing a full evaluation framework:

SOURCE:  McLaughlin, J. A. and Jordan, G. B. (1999). Logic Models: A Tool for Telling Your Programme’s Performance Story. Evaluation and Programme Planning, 22.

Existing frameworks outline how some types of evaluations are performed, for example, the NICE Evidence standard framework for digital health technologies classifies technology into evidence tiers and outlines the cumulative effectiveness evidence needed from tier 1 to tier 3b. The NICE framework also recommends an appropriate economic analysis depending on the potential financial commitment required.

The maturity of the technology and the body of evidence already generated for evaluation purposes

Health technology developers can approach evaluation teams at various stages of maturity of their technology. Differing evaluation activities are appropriate as interventions develop over time and move from the early to mid and late stages. These activities include . Evaluators may want to focus on the activities where evidence has not already been generated.

SOURCE:  World Health Organization. (‎2016)‎. Monitoring and evaluating digital health interventions: a practical guide to conducting research and assessment. World Health Organization.  https://apps.who.int/iris/handle/10665/252183. License: CC BY-NC-SA 3.0 IGO

Types of evaluations

There are two main types of evaluation – formative and summative.

A formative assessment helps to inform the design and development of an intervention so that it best suits the intention and objectives of the programme or provides direction on how to improve the intervention.

A summative assessment is conducted on established programmes at a fixed point or points, to measure the outputs and impacts of the intervention and determine whether it works or not. The intervention must be given sufficient time to achieve its goals, so this type of evaluation will measure the short and medium to long-term outcomes that result from the intervention.

Our approach at HEU

A systematic approach is needed to scope out the overarching aims and objectives of an evaluation, and to develop a robust evaluation framework, ensuring all evidence is generated in the full evaluation to facilitate the adoption of innovation and technologies into patient care.

At the Health Economics Unit (HEU), we develop logic models, evaluation frameworks and run full evaluations for our clients. Our approach and collaborative ways of working ensure we develop appropriate designs and that our bespoke teams have the right skills to deliver across the range of evaluation activities: usability, feasibility, efficacy, effectiveness and economic evaluation and implementation research. Our methods are also externally validated and/or peer reviewed.

If you’d be interested in working with us on an evaluation, whether you’re developing a new technology or considering implementation of a health care innovation, please contact us for further information.