Why public sector data needs to be transformed

Last month, I spoke at Holyrood Connect’s Public Sector Data Summit. From engaging with attendees and listening to their presentations, it’s clear that there is huge interest in the untapped potential of public sector data. The conference was a great opportunity to speak about the work Research Data Scotland (RDS) is doing to help unlock that potential.

RDS has been set up to enable faster, simpler access to public sector data for research, while - importantly - maintaining the high level of security of the current data access systems. Scotland already has excellent data on people, places and business, but barriers to accessing this data quickly and safely can delay valuable research.

The barriers to accessing data

When a researcher has a good idea for a research project, they have to undergo a lengthy process to gain access to the data. Based on recent interviews with our User Testing Group, it takes on average 6 to 24 months to access public sector data for research.

The data is sensitive, so it’s right that there are appropriate checks and balances in place, to ensure data is used only for the public good and appropriate privacy concerns are addressed, but the application process at present is more laborious and repetitive than it needs to be. Users felt that the guidance is unclear and incomplete; feedback mechanisms are slow, with over-reliance on research coordinators; and there’s a lack of accountability on who does what when.

Our interviews and wider discussions with data users and research coordinators alike have shown that there is a clear frustration about how long the whole process takes. Project delays mean that sometimes research funding or PhD studentships run out before the data is delivered, leading to wasted public benefit.

Researcher Access Service

RDS has been established to address these problems at the source – by simplifying the often confusing guidance, speeding up laborious processes, and digitising wherever possible. With the creation of the Researcher Access Service, RDS is looking to improve the end-to-end researcher journey.

Our initial work on the Researcher Access Service will improve the upfront information that is available to researchers; work to digitise the application process (including a portal for applicants to track the status of their application throughout); and introduce a risk-based triage approach to information governance, fast-tracking simpler projects that don’t require complex privacy considerations.

In order to bring the Researcher Access Service to life, we need to ensure that behind it, there’s a portfolio of data assets around people, places, and businesses held securely in the safe haven, ready to be accessed by researchers. To have the most impact, we want to maximise the use and re-use of this portfolio of datasets for research in the public benefit.

Portfolios of reusable data for research

In the past, researchers were granted permission to link datasets only for use in a single project. This means that after spending months, or possibly years, going through the arduous process of acquiring and analysing data, they had to delete their entire project data on completion. If another researcher wanted to access the same linked datasets later, they had to start the whole process again from scratch.

The COVID-19 pandemic gave us a clear example of just how valuable reusable data can be. In response to COVID-19, Public Health Scotland reviewed the data access request process and introduced a fast-track approach to speed up COVID research. The improvements reduced the time it took to access data for research to just one month.

Similarly, Administrative Data Research Scotland (ADR Scotland) is working to transform the way researchers best utilise public sector data. The ADR Scotland programme is led by the Scottish Government and Scottish Centre for Administrative Data Research. ADR Scotland is one of four national partnerships that make up ADR UK, alongside Office for National Statistics. Together they are transforming how public sector data is curated, accessed, and linked to facilitate impactful research that enriches the evidence base for policy to systematically improve the lives of the people of Scotland.

Working closely with the Scottish Government, RDS is working to support the delivery of the ADR Scotland programme from April 2023. For the Scottish Government, RDS will be updating and ingesting data to scale up the portfolio of de-identified, linkable datasets in the National Safe Haven.

A cohesive strategy

The system of making data available for research is very well established in Scotland, but it is fragmented. RDS is not the only organisation to recognise the improvements that can – and should – be made to the data access journey, but all of our partners and stakeholders are under significant pressure to deliver on their public functions.

RDS has been set up to take a bird’s eye view of the system, to act as an independent organisation, who can listen to all parties and suggest recommendations. We have a remit to provide systems leadership in bringing data organisations, universities, researchers, and public bodies from across the data landscape together, to work as a single system and transform the way data is managed.

The current process is sluggish and complicated, and valuable opportunities are being missed. But by working collaboratively, we will come together with a cohesive strategy to simplify systems, speed up access and, ultimately, improve lives.