Understanding healthcare data and the FDP

How the Federated Data Platform gives Palantir power over how NHS data is structured, processed, and consumed

When it comes to safety and transparency of data, the NHS FDP puts all its trust in Palantir, a company that holds contracts across the UK government, believes that data privacy, protection and security regulations are a significant risk factor to their business, and whose sales pitches have recently been shown to be repeatedly rejected by the Swiss government and army.

In this first article by NHS Analysts Together, data professionals share their understanding of how the FDP gives Palantir power over how NHS data is structured, processed, and consumed. It also aims to help the public and campaigners understand how healthcare data is generated, whom it goes to and what happens with it.

Despite living in the age of data and being aware of how our digital presence is tracked online and geographically, very few of us think about the data generated by our healthcare interactions and what happens to it. With the introduction of the NHS Federated Data Platform it is imperative that we, as citizens, understand who is using our health data and how. This has never been more important than it is now - NHS data has moved from being managed for the NHS by the NHS, to being managed and held by a private organisation that made its name and value from military AI and intrusive surveillance.

NHS Healthcare is a very ‘hands on’ industry. It relies on human interactions and a heck of a lot of doing: referrals, diagnostics, appointments for talking or prodding or poking, procedures, diagnoses, discharges, the list of actions is HUGE and every one of them generates data. To understand this data landscape we need to have a basic idea of which NHS organisations do what, so let’s have a look at that.

NHS Structure in a Very Brief Nutshell

We talk about The NHS as a single entity, however it is actually made up of thousands of individual organisations that work together to deliver healthcare. Each one is a separate entity with its own funding and generates its own data through interactions with patients and clinicians, and admin tasks. Because NHS organisations exist in their own right, the data generated at one organisation is not commonly shared with another.

NHS Data Collections in a Very Brief Nutshell

Different types of NHS organisations have different responsibilities when it comes to data collections and submissions. A majority of data returns are submitted to SUS (Secondary Uses Service), a repository of mandatory datasets. You can have a look at all the different data collections here to get a view of how broad the data collection scope is.

Many SUS datasets contain our personal and clinical data. This includes: names, addresses, date of birth, ethnicity, sex, spoken languages, and in some cases our sexuality and personal measurements such as weight and height. Submitting organisations have a legal obligation to submit the dataset in full to NHS England, regardless of any patient opt-out status. The data is then no longer the property of the submitting organisation and is processed and pseudonymised for sharing with Integrated Care Boards (ICBs) and other organisations who need to use NHS data to monitor activity, costs and performance. The shared data is duplicated across organisations where it is held in data warehouses that are all independently procured and managed according to local needs (data warehouses are repositories that are structured for efficient data recall and analysis).

It is important to note that there are also other local data collections to fill the gaps which SUS doesn’t cover, such as data shared between GP surgeries and ICBs. At every point the data is managed by the NHS, for the NHS - or at least this was the case until fairly recently.

The current NHS data landscape isn’t perfect. Duplications and local variations prevent smooth linkage across datasets and regions. To begin to tackle this sprawling nature of NHS data, the government decided to collate and standardise it by creating the Federated Data Platform (FDP). In October 2023 it was announced that the £330m contract to deliver the FDP was won by a consortium led by Palantir.

How the FDP works

Understanding the FDP isn’t easy. 417 of the 586 pages were redacted in the publicly published contract. Presentations to NHS data and analytics professionals about how the FDP works are vague, scant on detail and often accompanied by blurred, unreadable diagrams. However, now that the roll out of FDP is well underway, our network of data colleagues across NHS organisations is forming an understanding of how FDP works.

Piecing together our combined knowledge of FDP, we surmise that it uses an approach to managing data based on Data Mesh (Data Mesh is a type of data architecture and is not FDP specific).

Data Mesh principles ensure scalability and accountability - new datasets and organisations can be onboarded quickly and data has responsible owners. Essentially, Data Mesh is a method of managing data from lots of different sources by keeping ownership of the data with the data supplier and enabling access to it via a central platform. A very loose analogy would be to think of a traditional market - all the vendors show up and manage their own products and sales but they all sell in the same currency and abide by the same trading rules.

In principle, this is exactly what NHS data needs after decades of maintaining rickety data systems with sticking plasters and chewing gum instead of redesigning, overhauling and updating. However, as you might expect from a company specialising in surveillance, Palantir’s FDP solution strays from Mesh principles in a few important ways. True Data Mesh would give NHS organisations power over their own data, whereas FDP gives Palantir power over how NHS data is structured, processed, and consumed.

A key difference between true Data Mesh set ups and FDP is that organisations cannot see the code that shapes their data products. In contradiction to Data Mesh principles, control of the platform is heavily centralised as Palantir supplies the data engineers and manages the data infrastructure and logic. FDP uses the terminology of Data Mesh, but it doesn’t follow through with the autonomy or transparency associated with it. Transparency on logic is very limited and there is no ability to tweak products independently and reflect local variations in service delivery.

Going back to our market analogy, FDP’s markets are called “Instances” and data submitters control which Instances and people who access them can receive their data. At first glance, this appears very secure. However, we have heard from submitting organisations that the number of access requests from unknown Palantir data engineers makes it very difficult to track who within Palantir has access to their data for what reason. This mirrors a security risk raised by the US army with regard to lack of control of who is accessing sensitive material, and is a particular concern given the nature of NHS data and the potential to increase risk for both clinicians and patients. Think about how women’s data is being used for targeting of women and feticide prosecutions following the overturning of Roe vs Wade in the USA - this is happening despite protocols being put in place because of the untrustworthy nature of the companies who hold the data, and because law enforcement has accessed data.

A number of the bespoke local data flows we touched on earlier (the ones that cover the gaps in SUS), come from GP practices. You may be aware that there’s significant resistance to Palantir’s FDP from doctors and GPs, so getting their data into the platform is a challenge. To overcome this, ICBs are being advised to check the data sharing agreements they have with GP practices, and if the agreement does not specifically mention where shared data is to be held the ICBs can transfer the GP data to the FDP without informing the GP practice. This does not match the transparency and supplier control principles of Data Mesh.

Data Mesh focuses on flexibility, innovation and portability. As such it is platform agnostic. By contrast FDP has a strict ontology (data structure and relationships), limiting innovation and portability. The rules of the FDP are non-negotiable and the standards are defined by Palantir and NHS England - this is not federated, this is hierarchical. It also leads us to a vendor lock-in solution whereby moving the FDP from Palantir to another supplier would be extremely costly and complex, despite NHSE’s insistence that we are not locked in.

Shutting down the alternatives

Seemingly to combat resistance to the FDP, the government is also shutting down alternative options for NHS data management by closing Commissioning Support Units (CSUs) by December 2026. Currently, most ICBs use CSUs for their data solutions. Their closure, coupled with the government directive to reduce ICB running costs to £19 per head of population, means that paid-for FDP alternatives are simply not viable. In addition to this concerning strategic move which is resulting in thousands of redundancies, the NHS medium-term plan recommends that Trusts use the FDP for their data warehousing. This would give Palantir further unprecedented access and control over one of the most valuable and sensitive healthcare datasets in the world.

In summary

To reiterate, FDP isn’t all bad. The standardisation of data is helpful for analysts given the messiness of the current data landscape, and allowing apps to be built on the platform and easily disseminated is also welcome. Anything that supports improved accountability and data quality is a bonus too. But, the shadowiness of the contract, the drift from data mesh principles, the degree of control being handed over to a company with such a dubious ethical record, and the difficulty of exiting the contract once FDP is established should be concerning to us all, especially when so little consideration was given to alternative solutions which could have kept control within the NHS.

When it comes to safety and transparency of data, the NHS FDP puts all its trust in Palantir, which holds multiple contracts across the UK government and also believes that data privacy, protection and security regulations are a significant risk factor to their business.

We don’t know how you feel about that, but personally we don’t fancy Palantir, the company of a man who wants war crimes to be constitutionalised so he can make more money, in charge of the personal health data connected to us and the people we care about.

Subscribe to this Substack to be notified of our next blog post, coming soon.