Listening, learning, responding to concerns

CQC should undertake an evaluation that will identify and understand the outcomes and impact of CQC's response to this review.

Evaluation scope

The review was undertaken in line with agreed terms of reference. These terms were then used to identify 10 agreed aims of the review. These aims are an articulation of the intended outcomes and impact that is hoped will arise from CQC's response to the review. Throughout the report recommendations have therefore been mapped to these aims. These aims are:

1. The public, workers of services registered with CQC, and other stakeholders trust CQC to listen to and act on their feedback and concerns in an inclusive manner.
2. CQC has a culture, supported by effective policies, processes and practices, to listen to, act on, or respond to concerns raised by colleagues, including advisory and complementary staff, about CQC. This means staff feel safe to speak up and that speaking up is invited, welcomed, celebrated, listened to, and responded to well.
3. CQC has a culture in place, supported by effective policies, processes and practices, to listen to, act on, or respond to information of concerns about care from workers of services and others. It does this in a way that is free from institutional or interpersonal discrimination.
4. CQC works well with partners and providers when concerns about care are raised.
5. CQC's culture, processes including governance, decision-making and outcomes comply with, and look to lead best practice regarding the Equalities Act 2010, ensuring:
   - there is a clear understanding of best practice, where discrimination is identified, addressed and, wherever possible, prevented using anticipatory measures
   - the handling of concerns about CQC raised by colleagues, including advisory and complementary staff, are free from institutional or interpersonal discrimination
   - CQC makes reasonable adjustments for CQC colleagues and CQC applicants in a timely manner and in line with best practice.
6. Relevant CQC colleagues feel confident, skilled, empowered and supported to handle whistleblowing and information of concerns about care.
7. Relevant CQC colleagues feel confident, skilled, empowered and supported to respond to concerns raised by other staff, including advisory and complementary staff, about CQC.
8. CQC has a culture, underpinned by best practice policy, processes and practices, where staff, including advisory and complementary staff, feel empowered to make a meaningful and timely contribution during change to support improvement and transformation. This should include ensuring there is learning from, and an adequate response to, feedback from formal consultation and informal engagement.
9. CQC's appointment, contracting, engagement, deployment and disengagement processes relating to advisory and complementary staff are non-discriminatory, consistent with the values of CQC and ensure employment rights are maintained.
10. Relevant CQC colleagues feel confident, skilled and empowered to deal with employment litigation, including working with internal and external lawyers.

The scope of the evaluation should include tracking the recommendations to understand the degree to which CQC has taken the action expected of it to adequately respond to the review.

It should also seek to understand the impact of CQC's response to the review by investigating whether the 10 review aims have been met.

While the focus of evaluation should be on the degree to which CQC's response has achieved the aims, it should also look to understand why these have or have not been realised. This should include where recommendations have not been implemented. Where possible, the impacts of individual recommendations for action should be identified to inform future practice.

Where there is already a substantial change underway within CQC this should typically be out of scope of the evaluation. However, where the recommendations from this review significantly impact an ongoing change, their impact should be explored. If other programmes have a negative effect on the ability of the review aims to be realised, this should be recognised in evaluation findings.

Following the publication of this review, it is recommended that CQC bring in the Independent Review Panel at month 6 and 12 to receive their feedback on the progress of the evaluation and recommendation tracking. This would support in the independent overview of CQC's progress and build confidence in CQC's determination for both delivery of recommendations and continuous improvement and evaluation of the impact.

Timescale and reporting

There should be a report on CQC's progress on acting on recommendations 6 months after this review is published.

There should be an evaluation report setting out the outcomes of CQC's response to this review 12 months after this review is published. This should mark the formal close of this Listening, learning, responding to concerns review.

The full impact of the review is expected to take longer to deliver than 12 months. For example, cultural changes arising from changes to policy and practice will take time to emerge. Some elements of the evaluation should therefore be extended beyond the formal lifetime of the review to understand the degree to which the aims have been realised. Overall, it is expected that the evaluation should take 18 months to complete.   

Methodology

Evaluation should use mixed methods and include qualitative and quantitative analysis. It should consider involving understanding the feedback and experiences of:

• Workers in regulated health and social care services (surveys)
• CQC staff (surveys, focus groups, interviews, case reviews)
• Advisory and complementary staff at CQC (surveys, focus groups)
• CQC job applicants (surveys)
• Public (surveys)
• Partners and providers (interviews)
• Organisations that represent people who use services (interviews).

The evaluators should make sure that people contributing their feedback and experiences have their identities protected. Reporting of surveys, focus groups and interviews should ensure that no individuals can be identified.

Methods

Recommendation tracking

As part of our corporate performance CQC should track the delivery of the recommendations made as part of this review, alongside tracking of wider recommendations from our publications.

Monthly updates on progress and confidence should be provided to Regulatory Leadership Board, and at least quarterly updates to board sub-committees.

CQC has committed to its tracking recommendations in the corporate business plan and to provide an update in public board at the end of each quarter.

Surveys

Where surveys are already established, for example CQC's staff 'Pulse Survey', the evaluation should use this to support the suggested methods below.

• A survey of people who have provided feedback or raised a concern about care in the 6 months preceding the publication of this review, to provide a baseline. This survey could be repeated one year after publication of the review, for people who have provided feedback or raised a concern between 6 and 12 months after publication, to understand how experiences have changed. [Aims: 1, 3, 4]
• A survey of CQC staff within 3 months of publication of the review to provide a baseline across a number of aims. This survey could include questions for those who have raised a concern about CQC in the 6 months preceding publication of the review. This survey could be repeated with all CQC staff a year after publication of the review, to understand how experiences have changed. [Aims: 1, 2, 5, 8]
• A survey of job applicants who declare a disability to understand their experiences of reasonable adjustments. The survey could be live between 3 and 12 months after the publication of this review. Analysis could be completed in 3 month tranches to understand if experiences change over this period. [Aim: 5]
• A survey for all advisory and complementary staff exiting CQC from 6 to 12 months after the publication of this review, to help identify whether processes are fair and consistent with the values of CQC and ensure employment rights are maintained. This could be sent 12 months after the publication of the review. [Aims: 2, 5, 9]

Focus groups

• Focus groups with CQC staff to understand whether they feel safe to raise their concerns between 9 and 15 months after the publication of the review. [Aims: 1, 2, 3, 4, 5, 8]
• Focus groups with CQC colleagues who have requested reasonable adjustments to understand their experiences, 6 to 12 months after the publication of the review. [Aims: 2, 5, 8]
• Focus groups with those responsible for handling whistleblowing and information of concern about care, between 6 and 9 months after the publication of the review. [Aims: 2, 3, 6]
• Focus groups with those responsible for responding to concerns about CQC, including People Directorate colleagues, Complaints Team, and the guardian(s), raised by CQC colleagues including advisory and complementary staff(s), between 9 and 15 months after the publication of the review. [Aims: 2, 5, 7]
• Focus groups with advisory and complementary staff about their appointment, contracting, deployment, between 6 and12 months after the publication of the review. [Aims: 1, 2, 3, 4, 5, 9]

Interviews

• Interviews with partners and providers about how well we have worked with them when concerns about care are raised, between 12 and 15 months after the publication of the review. [Aims: 4]
• Interviews with organisations that represent people who use services about how well CQC listens to and acts on feedback and concerns in an inclusive manner, between 12 and 15 months after the publication of the review. [Aims: 1]

Case reviews

For CQC employment tribunals concluding between 6 and 15 months after publication of review there could be case reviews by legal and HR colleagues. These case reviews could assess whether the learning from Mr Kumar's tribunal case has resulted in appropriate changes to CQC's culture, policies, processes and practices relating to employment litigation. [Aims: 4, 9, 10]

Analysis of CQC data

• Leading indicators should be identified and tracked.
• Analysis of whether CQC has acted on information of concern about care, and how this varies by protected characteristics. This could be undertaken at 6 months and 12 months after report publication. This will look at month change and comparison to the original analysis undertaken for this review. [Aim: 3]
• Analysis of whether CQC has responded to or acted on information of concern about CQC, and how this varies by protected characteristics. [Aims: 2, 5]
• Analysis of proportion of reasonable adjustment requests met and the length of time to implement them. [Aims: 5]
• Analysis of the public awareness and sentiment tracking survey. [Aims: 1]

Evaluation reference group

A reference group should be set up to help ensure evaluation findings hold CQC to account in achieving the aims of the review. This should include people who have been impacted by the areas of focus within the review and those responsible for implementing the recommendations.

Limitations

There are limitations in the evaluation being able to reach those who speak up or raise concerns. This is particularly the case where concerns are raised anonymously, or contact details are inaccessible or not up to date. Hopefully action to implement this review's recommendations will help mitigate this risk.

Some quantitative measures may not reflect impact at the formal close of this review, 12 months after the review publication. Consequently, the evaluation should continue for up to 6 months beyond the end of the review to understand the longer-term impact and to support ongoing improvement. However, some of the impacts may still not have been realised by the end of the evaluation period, which could limit evaluation findings.