User testing survey analysis - how do guideline users use and interact with the Sentencing Council’s website? Part 1 - Summary

1. Summary

1.1 Introduction

The Sentencing Council for England and Wales (‘the Council’) is an arm’s length body f the Ministry of Justice (MoJ). The Council was set up to promote greater transparency and consistency in sentencing, while maintaining the independence of the judiciary. It has responsibility for developing sentencing guidelines and assessing their impact on sentencing practice, among other things.

The Council committed to explore how guideline users access, use and interact with the sentencing guidelines on its website in its strategic objectives 2021-2026 in November 2021. In order to do so, we decided to conduct research via a two strand approach. The first strand was a survey distributed by the Council to guideline users, the results of which are outlined in this report. This analysis informed the second strand of the research which was externally commissioned to the Behavioural Insights Team (BIT). BIT carried out in-person observations, virtual usability testing and semi-structured interviews with sentencers. A separate report has been published by BIT, which can be accessed here.

1.2 Methodology

This research comprised a survey designed to capture current practice and use of the guidelines (offence specific and overarching) as well as features and tools available on the website. Offence specific guidelines relate to particular offences or sets of offences, while overarching guidelines provide guidance on cross-cutting areas that can be applied across a range of offences, for example domestic abuse and sentencing of children and young people. The Senior Presiding Judge gave approval for judicial participation in this project.

The survey was completed by guideline users, including magistrates, judges and other court users such as legal advisors. Users were asked to complete a questionnaire which was advertised through a bulletin on the homepage on the judiciary’s intranet. Legal advisers were notified of the opportunity to respond to the survey via a message sent to their membership body, the Justices’ Legal Advisers and Court Officers’ Service (formerly Justices’ Clerks’ Society). The survey was hosted on SmartSurvey; please see Annex A.

The survey was voluntary, open to all sentencers and relevant court users between 1 September 2022 and 2 October 2022. It contained a mixture of multiple option and free text response questions. Where appropriate, we conducted thematic analysis of the free text responses. In some cases, responses to these free text questions either duplicated information already provided by the respondent in the multiple option section of the question or did not directly answer the particular question. In these instances, responses were removed to avoid duplication or inclusion of non-relevant information.

We received a total of 1,731 responses, most of which came from sentencers (95 per cent) as opposed to other users of the guidelines Due to the self-selecting nature of this survey research and only a small proportion of the overall guideline user population responding, the findings of the survey are unlikely to be representative of the whole guideline user population. In addition, magistrates made up 90 per cent of respondents, therefore this group is overrepresented in the sample (according to 2022 figures released by MoJ, magistrates make up approximately 78 per cent of the overall sentencer population and judges 22 per cent; note that these estimates do not include judges whose jurisdiction is outside of criminal sentencing). Despite this, the responses have given crucial insight into the way in which sentencers and other users use the website.

Early survey findings relating to the guidelines themselves were shared with BIT to inform the second strand of research. Sharing the findings allowed us and BIT to agree the scope and areas of focus for their research, which included in-person observations using hypothetical scenarios, qualitative interviews and virtual usability testing with sentencers.

1.3 Findings

  • Experience of using the guidelines: almost half of the survey respondents had been using the sentencing guidelines for between 11 and 20 years, and just over half had been using the guidelines for under 10 years. Around 40 per cent of respondents had been using the guidelines for under 5 years
  • Accessing the guidelines: respondents predominantly found (79 per cent) and used (69 per cent) the online version of the guidelines via the Sentencing Council website. A small number found (24 per cent) and used (18 per cent) them via the Sentencing Council app (which is restricted for use on iOS devices only). Use of the app decreased when looking at a subsample of judges compared with magistrates. A small number of magistrates also mentioned that they would seek advice from colleagues or court officials, or via a search engine or a third party app (UK Court Manager). Some respondents mentioned a shortcut to the website available on eJudiciary (the intranet homepage for court users and the judiciary) or the desktop of court laptops which links directly to the guideline search page.
  • Accessing the guidelines – hardware: the majority of respondents (65 per cent) accessed guidelines via a Windows laptop and 21 per cent accessed them via an iPad. A small number accessed them via an android tablet, phone or iPhone (around four per cent). Around six per cent accessed guidelines via desktop computer, and those remaining stated that they used an Apple laptop or used two or more devices to access the guidelines.
  • Steps in the guideline: generally, sentencers reported that they were more likely to refer to the earlier steps of the guidelines. This may be expected as these sections contain culpability and harm, aggravation and mitigation (factors of the case that may increase or decrease the sentence), as well as the sentencing table. Although no direct reasoning was sought for this question, this may be explained in part by the free text responses to other questions, in which a theme of familiarity with the content of guideline sections arose: if sentencers felt they were familiar with the content of a section of the guideline, they would not revisit this section for each case for which it may be applicable. This is to be expected as the later steps of the guideline tend to be similar across guidelines, whereas the earlier stages vary depending on the offence.
  • Applying aggravating and mitigating factors: over 70 per cent of respondents reported that they applied aggravating and mitigating factors which are not listed in an offence specific guideline at least some of the time. A very small number did not apply any factors which had not been listed in an offence specific guideline. This finding could be expected as all step two factors in offence specific guidelines are non-exhaustive.
  • Application of overarching guidelines: respondents were asked how often they access and then apply overarching guidelines, where relevant to a case. Across all guidelines, responses varied, and no consistent pattern occurred. In general, the Sentencing children and young people guideline was referred to and applied the least often. This is perhaps unsurprising as cases involving children and young people make up a small proportion of the overall court workload and the majority of respondents specified that they did not sit in the youth court. For example, recent MoJ statistics (2021-2022) state that of 88,600 first time entrants to the criminal justice system, 9 per cent were children aged 10 to 17 years. A key theme was a feeling of familiarity with the principles contained within the overarching guidelines, which meant respondents did not feel it was necessary to access the guidelines for every case for which the principles may be relevant. This was overwhelmingly prominent for the Reduction in sentence for a guilty plea guideline, with which sentencers felt most familiar and was commonly understood. However, sentencers stated that they would refer to the guidelines if they had been notified of a change or there was a point that needed clarification.
  • Sentencing in the absence of an offence specific guideline: when sentencing in the absence of an offence specific guideline, 96 per cent of respondents relied on legal advisers or counsel submissions (it should be noted that legal advisers only sit in the magistrates’ court – this finding is therefore not applicable for the Crown Court), 59 per cent relied on guidelines for similar offences, and 48 per cent and 45 per cent, respectively, relied on the General guideline (published for use in the absence of an offence specific guideline) or previous experience of similar cases. A small number relied upon Court of Appeal case law. In free text responses, 18 people stated that they would consult literature on sentencing (‘Banks’, ‘Archbold: Sentencing Guidelines’, ‘Thomas’ Sentencing Reference’, and similar), and 23 stated that they would seek advice (from colleagues or court officials such as the legal adviser, Crown Prosecution Service (CPS), advocates or counsel).
  • Imposition of custodial sentences and community orders: respondents were asked if they were aware of the dropdown explanations, available in step two of the guidelines, which explain the principles behind the imposition of custodial sentences and community orders. Just over 80 per cent were aware of them and referred to them; 8 per cent were aware of them, but did not refer to them; and 8 per cent were not aware of them. Of respondents who were aware of them and used them, the majority found them helpful, with only a small number stating issues around difficulties accessing the information, the length of the explanations and their relevance. Those who did not refer to the dropdown boxes stated familiarity with the content or had found the explanations unhelpful previously. Others mentioned a lack of time, that they preferred to use a third party app, or had forgotten the dropdown was available.
  • Expanded explanations: respondents were asked if they were aware of the ‘expanded explanations’ available for certain aggravating and mitigating factors, and whether they referred to them. These explanations describe the type of issues to consider when applying the factors and at the time of the research could be accessed via dotted lines underneath them (these have now been changed to dropdowns). The majority of respondents (76 per cent) were aware of the expanded explanations and referred to them. Of those who were aware of them but did not refer to them (6 per cent), 49 per cent stated that they had accessed the explanations previously and understood the principles. Respondents who were aware of them and used them, overwhelmingly found them helpful, with just a small number mentioning a lack of court time and familiarity with the content as reasons why they were not found helpful. Eighteen per cent of respondents were not aware of the explanations.
  • Drink-driving calculator: (magistrates and magistrates’ court users only). Although 87 per cent of respondents were aware of the drink-driving calculator, only 63 per cent had used it. A smaller proportion, 13 per cent, were not aware of it. Of those who did not use the calculator, this was because: the legal advisor leads on the calculation; a third party app is preferred; issues have been experienced with inputting, formatting and/or incorrect calculations; or a preference to calculate the disqualification manually.
  • Pronouncement card builder: (magistrates and magistrates’ courts users only). Respondents who were aware of it but did not use the builder (66 per cent) outweighed those who used the tool (29 per cent). Only a small proportion of magistrates/magistrates’ court users were not aware of the builder (five per cent). Of those who did not use the function, respondents most commonly noted that the tool is primarily used by the presiding justice (magistrates normally sit in a group of three when in court, with one being the presiding justice, who speaks in court and presides over proceedings, and two ‘wingers’ sat on either side). Other reasons for not using the tool were: that it is not user friendly; the presider prefers to tailor the pronouncements for each defendant to ensure the content is understood; that a third party app is preferred; and a lack of time.
  • SentencingACE tool: (Crown Court judges and users only). Of the small number of Crown Court judges and Crown Court users (64) who responded to the survey, just under half (45 per cent) stated that they were unaware of the SentencingACE tool (this is designed to allow judges to check that the elements of the sentence they wish to impose are lawful). Thirteen per cent were aware of it and regularly used it, and just over 42 per cent were aware of the tool but did not use it. Reasons cited included: lack of time; the tool offers little assistance; and that the tool is too cumbersome. A small number noted difficulty locating it.
  • Sentencing Council website: Overall, respondents (46 per cent) noted that they found it fairly easy to use the website to access and use the guidelines. However, a few noted confusion between the search function on the homepage and the search function for finding the guidelines. Due to this, respondents noted that they had difficulty finding pages of the website that they had previously been able to find or were looking for, for the first time.
  • Offence specific guidelines: Just under half of respondents (45 per cent) also found it fairly easy to find offence specific guidelines. Despite this, many comments were left in relation to some difficulties finding these guidelines. There was a focus on using the search function, with many saying it is difficult to use or that it is not intuitive. A high proportion of respondents expressed frustration around using the search function for finding offence specific guidelines. Difficulties were also noted with the title of the guideline not matching the charges on the court list sheet which is used in court and which could serve to increase the time to find a guideline. There was also some overlap between comments in relation to finding overarching and offence specific guidelines on the website, again primarily around difficulties using the search function, which was described by one respondent as “the weakest link in the guidelines.
  • Navigating guidelines: Respondents generally found it easy to navigate through the different steps in the guidelines on the website. A small number noted frustration with scrolling up and down to find the appropriate information.

Analysis of this survey has improved understanding in how users access, use and interact with the Sentencing Council’s website. It has identified areas which could be improved to impact the usability of the tools, functions and guidelines available, such as the search function and fine calculator. In response to these findings, as well as the recommendations set out in the BIT report, the Council is considering a number of changes and improvements to improve the experience of guideline users.