Insights ¦ The Evaluation of the M365 Copilot Pilot in the Department for Business and Trade

Key Take Aways

1. The evaluation of Microsoft 365 Copilot within the Department for Business and Trade indicates high overall user satisfaction, with 72% reporting they were satisfied or very satisfied during the pilot.
2. Neurodiverse colleagues and non-native English speakers report notably higher benefits and satisfaction, highlighting Copilot’s accessibility benefits.
3. Time savings were observed predominantly in written tasks, such as drafting, summarising, and transcribing meetings, although the magnitude was generally small.
4. Control group colleagues did not observe productivity gains from Copilot, and self-reported time savings were not conclusively linked to increased departmental productivity.
5. The primary use cases included summarising research, transcribing meetings, and composing emails, with less frequent use for scheduling, code review, and generating images.
6. Almost 80% of respondents found Copilot useful, with a Net Promoter Score of 31, reflecting good user endorsement of the tool.
7. Satisfaction was strongly linked to self-led training; those undertaking at least two hours of independent learning expressed higher overall satisfaction.
8. The tool’s outputs generally were accurate, but hallucinations and inconsistencies in quality assurance were noted, necessitating manual review.
9. Attitudes towards AI, including concerns about environmental impacts and ethical considerations, moderate adoption and use within the department’s culture.
10. Specific roles with heavy administrative tasks and handling large volumes of documentation saw the greatest benefits, whereas roles requiring high contextual awareness experienced more limited gains.
11. Observed tasks confirmed that M365 Copilot can improve the speed of producing presentations and summarising reports, but some tasks like data analysis could be slower or less accurate.
12. Limitations identified include a low response rate, potential biases in self-reporting, and small sample sizes in qualitative and observed tasks, warning against overgeneralisation of findings.

Key Statistics

• 72% of respondents reported being satisfied or very satisfied with M365 Copilot.
• The Net Promoter Score was 31, categorised as a good score.
• 80% of respondents found Copilot conceptually useful in their daily work.
• 43% of users neither identified nor experienced hallucinations; 22% identified hallucinations.
• Mean daily active user rate during the pilot was 64%; peak was 68% in mid-October.
• Approximately 1.14 M365 Copilot actions were taken per user per day on average.
• The most common use cases included writing emails (1,679 instances), summarising emails (153), and transcribing meetings (234).
• Overall satisfaction varied by task, with slide decks and scheduling recording only 54% and 57% satisfaction levels respectively.
• Training correlated with higher satisfaction; at least 2 hours of self-led training increased likelihood of positive outcomes with a P value of 0.00005252.
• 36% of respondents reported that quality assurance for scheduling outputs took more time or was not comparable.
• Observed task data indicated that M365 Copilot improved speed for PowerPoint and report summarising but was less effective for Excel data analysis and generated outputs of variable quality.
• 30% of diaries were collected from volunteers, with remaining samples stratified to represent the department.

Key Discussion Points

1. High overall user satisfaction signals the potential of M365 Copilot to enhance administrative efficiency and individual confidence, especially for neurodiverse and non-native speakers.
2. The impact on productivity remains inconclusive; self-reported time savings have not demonstrably translated into departmental performance improvements.
3. Training, especially self-led, is critical in maximising satisfaction and perceived value of Copilot’s capabilities.
4. The tool performs best in written communication and summarisation tasks but is less reliable for complex data analysis and scheduling.
5. Hallucinations and output accuracy are pertinent risks, necessitating diligent quality assurance and manual review by users.
6. Attitudes towards AI, shaped by ethical concerns and environmental impact perceptions, influence adoption levels; some colleagues remain cautious.
7. The accessibility benefits are significant, with neurodiverse and disabled users reporting empowerment and reduced cognitive load.
8. Role-specific benefits vary; roles with heavier documentation and administrative duties benefit more than those requiring nuanced judgement, such as legal and policy roles.
9. Limits in the deployment include a response rate of only 32%, potential volunteer bias, and small qualitative/observed sample sizes, constraining broad generalisability.
10. Observational analyses confirmed that, for certain tasks, using Copilot accelerates output delivery, though some outputs lacked quality or accuracy, highlighting the importance of manual review.
11. Participants expressed a desire for more transparency regarding the environmental impacts of AI and for future tools with enhanced automation and integration.
12. Overall, the evaluation underscores that while M365 Copilot shows promising benefits, careful implementation, ongoing training, and risk mitigation are essential for sustainable success in a senior management context.

Document Description

This article presents a comprehensive evaluation report of Microsoft 365 Copilot pilot within the Department for Business and Trade. It details the methodology, quantitative and qualitative findings, and insights into user satisfaction, productivity impacts, and ethical considerations. Tailored for senior managers, it delivers insights on how the AI-powered assistant is shaping administrative workflows, highlighting opportunities for strategic deployment and areas requiring caution. The report also discusses operational challenges, role-specific benefits, and critical limitations, serving as a foundational document for future AI integration in government or highly regulated sectors.