The power of sharing data beyond a pandemic

In my former role as Scottish Government’s Chief Statistician and joint head of the Covid Analytical Team, I saw first-hand the importance of having data flowing at pace to the people who needed to design and implement new policy and understand how they were working quickly. 

With fantastic support, I established the Scottish Covid-19 Data and Intelligence Network to create the space where we could get data sharing happening, and to make sure this was done legally, ethically, and with the support of the public. The network identified shared challenges across the public sector, and brought people together to solve those challenges. These were both topic based challenges like how we could best identify at risk groups using data, and policy challenges such as establishing a data ethics approach for public bodies. 

I’ve mentioned before that one of the most important positive legacies of COVID-19 is the recognition that high quality statistics and analysis don’t just happen – that we need a nurtured infrastructure to make it happen. While Scottish Government saw the need for an organisation to transform and simplify that infrastructure by announcing the establishment of Research Data Scotland back in 2019, the data requirements of the pandemic brought home to many the need to quickly realise its ambitions.  

Data linkage enables new insights 

There were some outstanding examples of where anonymised datasets were linked together and used to provide valuable new insights that shaped the Covid response. Projects like EAVE II proved to be an invaluable source of information in tracking the COVID-19 pandemic and vaccine effectiveness across Scotland. The project achieved near real-time updates during the pandemic using a rich dataset of 5.4 million patient records – everyone registered with a GP in Scotland, which was around 98% of the population at the time.  

It also allows us to understand how implemented policies have affected citizens, such as this study on Fixed Penalty Notices in Scotland, and how they were used in policing during lockdown periods.  

Incomplete datasets only provide part of the picture 

As Covid-19 infections spread in the spring of 2020, early reports emerged from England that ethnic minority communities were being disproportionately impacted by the virus. Answering this question turned out to be really tricky as the data available within health data systems on ethnicity was not complete enough, potentially biasing any statistical analysis.  

However, there was data on ethnicity and other demographic characteristics elsewhere – in the population census and on other public sector data systems. By bringing together expertise from around Scotland, we were able to bring together a deidentified equalities research dataset that combines data from a range of places and enables research on demographic and intersectional issues. 

Improving data on demographics is vital for policy development that can consider impact across different groups in society. The equality research dataset is only part of the solution here and I know that many people are setting up arrangements to systematically gather good equalities data, particularly through the Scottish Governments Equality Data Improvement Programme. 

Quicker and simple access to data 

With access to data playing such a crucial role in pandemic decision-making processes, the need was clear for a long-term solution to using our public sector data for policy making purposes. This is where Research Data Scotland comes in. Our mission is to simplify access to public sector data to advance the health and social wellbeing of people in Scotland. To do this, we’re working to provide a single point of contact for safe, secure and effective access to secure datasets.  

We already have the data within our public sector organisations that makes it possible to gain insights that can help us make crucial public health decisions.  

While we are making good progress in unlocking these valuable assets, there remains lots of valuable data held by public bodies that is scattered across different systems, locked away in formats that are difficult to access, which can get in the way of linking together important pieces of data. This is still a real barrier for person-centred research, policymaking, and improving our public services. Our Researcher Access Service is a significant first step. This is a brand-new service that will, for the first time, provide an end-to-end digital process for researchers and data users to access secure data for the public good.   

The COVID-19 pandemic showed us the benefits of using Scotland’s data for public benefit. My role is now to drive the changes needed to get to a place where data driven approaches are systematic rather than a limited set of examples where data really does save time, money and lives. 

So, if you are an analyst or researcher, or you work in an organisation that has data about Scottish people, places or businesses, please reach out. Your data, your insight and analytical expertise can be part of our covid legacy in improving people’s wellbeing and creating a more equal and just society.