What will Data Exchange (DX) mean to Schools?

Deliberately following on from an earlier article: The Data Landscape is Changing – Are we Ready for it?, which received a mixed, yet interested reaction (On Edugeek, here), as well as a follow up from like-minded colleague Matt Smith (His blog, here), I have sourced/published as much information taken directly from the Department for Education, and any events or documents produced by them, as possible and summarised this here.

Reactions to The Data Landscape is Changing were exactly what the article was designed to illicit: surprise, anxiety, perhaps even a little fear. I’m not surprised a bit by this, which is why the following information is taken directly from a number of published presentations and documents outlining Data Exchange (DX) and the Schools Performance Data Program (SPDP) vision and scope, which is the Department’s own research and views on how DX/SPDP will be transforming data collection, process and sharing across the sector, and how this Data Transformation Program (DTP) will greatly benefit all within the sector, from pupils to schools to LAs to the sector as a whole.

This begins with the Analytical Review commissioned by the Secretary of State, completed by Roger Plant and presented to the Department, which was then followed by the identification, scoping and design of SPDP and DX (an overview of the process, there were a lot of other steps and people involved!). The elements are taken from a range of sources over time from Spring to December 2013 (so some specific/technical details will have changed since then, but the vision remains the same).

Below too are two presentations containing a wealth of plans and visions from the DfE. The first in particular from Summer 2013 explains in much more detail the outcomes from the Analytical Review as well as a detailed look at how a similar program is rolling out in Australia (Learning from International Experience). I believe this is a CRITICAL analysis showing how such a project can and is being rolled out in a developed education system backed up by government, and a good testimonial where otherwise would be impossible to show (as nothing like this has been done in the UK).

Both have been edited for length and for irrelevant slides.

Presentation: Summer 2013 DX

Presentation: November 2013 DX

Elements are extracted from various documents, all publicly available (links included where appropriate). Some elements are edited for length only. Sources publicly available on ContractFinder.


Analytical Review

The Secretary of State commissioned an independent review of the methods by which the Department collects information for analytical (accountability, funding, policy development, etc.) purposes. The terms of reference for the review asked that a system be found where only data essential to the DfE and to a devolved self-improving system is collected. Such a system needs to be efficient, trusted and must maintain appropriate standards of quality.

The review concluded that technology in the DfE used to capture data is not in line with current industry standards and is almost ten years old. Whilst the DfE operates reasonably efficiently given the constraints of the supporting systems, the systems themselves drive considerable inefficiencies in data collection and use. These raise the costs incurred by the DfE, schools, colleges and children’s services, and also restrain the development of a self-improving system. In time, current systems will become too expensive to maintain and not fit for purpose in an increasingly autonomous system.

It recommended that DfE move from their current technologies, which manage the bulk annual upload of large files, to a messaging system that responds to business events within front end systems to inform all who need the information, when they need the information.

DfE Data Transformation Programme: Improving the way education data is moved, stored, accessed and used


The Data Transformation Programme consists of 3 strands: The Data Exchange Project (DE) has taken the steer from the Analytical Review about replacing collection technologies with near-real time messaging, and worked up a technical requirements, options appraisal and strategy document. It is now developing the detailed specification and products prior to submitting a formal business case.

The School Performance Data Programme (SPDP) will deliver a data warehouse to rationalise storage and improve the processing of data, and a web portal to streamline access to key Departmental data and improve the user experience.

Information standards development & implementation. Common standards are key to making it easy and efficient to share data quickly, and reduce the amount of effort needed to process it. Both SPDP and DE will build on the Information Standards Board (ISB)-controlled data model that has been developed with and for the entire Education, Skills and Children’s Services sector.

The projects are at different stages in their lifecycle, and as we bring them together into one programme the intention of this note is to set out a shared end to end vision which is agreed by all parties, which then can be tailored for sharing publicly.

The current state of play

The Department for Education (DfE) collects and holds a wealth of data about learning providers (and the children who attend them) and Local Authority (LA) provision of children’s services, which it obtains by running a number of large data collections from schools, local authorities and awarding organisations. This data is used for a range of funding, accountability, policy development and operational purposes.

The arrangements by which data is collected, processed, stored and published have evolved over a number of years and now operate in inefficient silos on outdated legacy systems that are managed by separate teams and often require significant investment to develop in response to new or changing policy needs. Changes to data collections are difficult and take up to 18 months to implement, bulk collections require more front line administration than desirable during which we miss opportunities to validate at source, and our processes limit ambitions due to poor timeliness and infrequency. Data is stored disparately in discrete systems and databases that are difficult to join up and analyse. The data we publish is made available through a number of different websites, with similar information often presented in multiple places in slightly different ways, and using separate user registration systems, which results in a fragmented and frustrating experience for users.

Current - DTP

Taken together, these mean that we are slower than we would wish in developing evidence, publishing data and statistics, and making detailed data available to support transparency initiatives. We and the wider education and children’s services sector also do not fully exploit the power of data to inform the content of individual learning / service provision, and help those responsible deliver top quality services, as set out in the Analytical Review.

Major opportunities have arisen to radically improve this data landscape, which the DfE is moving to capitalise on. The contracts which underpin DfE’s storage and analysis of school and pupil performance data expire in 2013-14 and need to be replaced. The DfE Analytical Review set a new direction for how data should be collected. There is widespread appreciation within the education sector and across government of the benefits that will arise from adopting common information standards to minimise the effort taken to gather data and maximise reuse. And government more generally is driving to open up its data and improve the citizen experience.

The Vision

Our vision is for richer, more accurate data to be available quickly in accessible and usable forms, in order to enable others to drive up the quality of education and services received by children.

The Data Transformation (DTP) Programme will streamline, reorganise and modernise the current arrangements for how the DfE manages information about schools, LAs, pupils, workforce and children’s services. The movement, storage and utilisation of data must be made simpler, clearer and faster for its diverse set of users.

Future - DTP

We are designing the new services with flexibility and scalability in mind so that they can easily respond to and cope with policy changes. They will be developed iteratively, ensuring that they are delivered cost-effectively and align with what users want. Making more data available in a more usable form will allow third parties to develop applications and data tools that will benefit schools and the sector more widely.

Running the new services is expected to drive out significant benefits and quantifiable savings, including operational efficiencies in both DfE and the wider sector as a result of reducing the manual effort required to manage data processes. At the same time, these services will facilitate innovative use of data by teachers, social workers, policy makers and parents alike, and so shape the delivery of education and children’s services.  It will be the vehicle for the exchange of curriculum experience and outcomes data to support the ‘evidence based teaching’ philosophy set out within the Analytical Review.

1. Moving education data around more efficiently (Data Exchange).

We need to move away from bulk data collections to one where data can be moved regularly, with minimal manual intervention. A business process (such as registering a pupil, a child being placed for adoption, or noting an absence from school) should trigger pre-determined data movements to keep authorised users of data as up to date and informed as they need to be. This could mean finding out about something within minutes when necessary, but more typically within hours. Data should have the potential to be ‘pushed on change’ or at defined times or to be pulled. Once a series of data movements have been set up, they should have the potential to continue to flow with no further effort required.

Currently, schools have to send the same information to several places: a school to which a pupil is moving; the local authority; and DfE potentially. This is inefficient.  Data should be made available to all who need it, when they need it, without the need for the provider of that data to package it up in a bespoke fashion for each receiving system.

MIS Vision - DTP

We anticipate each and every school and LA, and ultimately every learning establishment and broader consumers of education data, being plugged into an exchange via their Management Information System. All these organisations which plug into an exchange are called ‘end points’. Each will have users who are appropriately authorised with permissions appropriate for their role. We envisage data movements being managed by a central hub and end points, together controlling the content, frequency, direction, and efficient loading of data movements.

The Department’s data warehouse (see section 3) will be a significant end point within the architecture. Indeed, it is such a big consumer that taking every opportunity to integrate the exchange architecture with that being procured under SPDP will maximise our chances of achieving quality and value for money.

End points currently have, and will continue to have, requirements to securely move a variety of types of information to each other.  Therefore we anticipate that the data exchange should be able to handle a variety of formats for moving data around the sector.

The data when it moves will be in a common format (the data model) but the agents or adaptors will manage the translation needed for each end point. To govern new processes, we expect significant front line involvement as we create a group responsible for overseeing that data movements are proportionate and based on business need.

To move data around in this way, it needs a data model that all end points are aware of and can communicate with.  This is where the Information Standards Board’s Enterprise Data Model (aka Business Data Architecture) is a key enabler for the vision.

2. Efficient management, storage and processing of data – SPDP Data Warehouse)

Data will be stored in a standardised data set in a single, secure data warehouse, significantly reducing the dependency on a number of separate specialist contractors and stand-alone data stores.  Having the data in one place will simplify arrangements for governing, managing and accessing data; reduce duplication of systems and effort; enable many processes to be made more efficient; and strengthen quality assurance so that there are fewer errors in departmental outputs Data items will be stored at the lowest possible level of granularity, increasingly based upon the new ISB Enterprise Data Architecture, providing maximum capability to respond quickly to changing policy and operational needs for new analyses and reports.

Data will be aggregated into the key reports and measures required by departmental outputs, starting initially with those relating directly to school and pupil performance which support the accountability system. The data warehouse will be built to scalable design principles, able to hold other data gathered by, or of interest to the Department, so that it can continue to grow and develop with relatively little further investment and become a corporate asset that supports operations across the whole DfE.

3. Improved user experience – SPDP Portal

Data will be made available to users via a single portal that brings together and enhances the functionality currently provided in a range of separate websites operated by DfE and Ofsted.  Accessed through gov.uk, it will be much easier for users to find the data they want and utilise it more effectively. Explanatory information to support user interpretation of the data will be made more consistent and simpler to locate.

New data visualisation tools – such as dynamic tables, charts and maps – will help users to more quickly interrogate and query the data, and drive greater data analysis to suit individual needs. The portal will provide both public, open access and secure, authenticated access to data.

It will enable parents, the public, schools and other education establishments, local authorities, educational practitioners and other interested organisations to access the data they require via a single website.  It will facilitate the manipulation, analysis and display of the data according to the needs of the users, and will extend access to include touch-based, mobile devices.

An Application Program Interface (API) will support the development of user friendly tools by third parties. The portal will be developed in line with Government Digital Service principles, helping to future-proof it and ensure that we can respond more quickly to changes in customer and user requirements than is currently possible.

4. Using a common language (Information standards)

Key to achieving an efficient solution is an agreed way of describing the detail of how data should be presented for movement.  The Information Standards Board (ISB) is the system-wide authority for all Education, Skills and Children’s Services information and data standards. Our Programme’s relationship with them is that we utilise ISB to provide or develop standards relevant to the data we need to move.

Fundamental to achieving the Vision is designing data communications so that items of data will always be transferred in consistent ways. As standardisation increases across software systems, so the costs reduce.  Data should come into the exchange in an agreed format, and flow out of it in this format.

Transformations to get systems ‘talking to each other’ must happen within the sending / source end points (e.g. a school MIS) or associated agents / adaptors. We also seek to develop and implement future-proofed data standards so that there is minimal impact on SPDP and DE from any new policy.  New data demands from the new policy can then be dealt with more effectively and monitoring new policy changes can be carried out more efficiently. Developing the data architecture in a manner which maximises the realisation of benefits is the role of the ISB.

As well as developing the standards for an ever broadening data model to service the exchange, Education Data Division and ISB will work with the sector and industry to manage a migration to ISB standards which occurs to achievable timescales.


The scope of the Data Transformation Programme is to improve the capabilities for sharing and utilising data within the education sector. The programme aims to provide the capability for the rapid transfer of information between organisations within the sector and to reduce the manual burden in moving data around. The scope also includes the provision of a central repository of information which can be accessed by those who need it.

Both users within the education sector and members of the public will be able to access the information that is shared as part of the programme. These capabilities will allow the Department to receive information from across the sector, process it, and publish it for access within hours in some cases.

A number of scenarios have been identified which fall outside the scope of DTP, including: the transfer of information between systems within an organisation, for example to maintain common data in separate systems within a school in a consistent state on a sub-second timescale; the transfer of information between schools working collaboratively, for example to move in-lesson attainment data captured in an interactive learning environment from one school to another during the lesson; as a rapid monitoring and alerting systems (for example to alert the Local Authority children’s services immediately when a learner that is being monitored does not arrive for school).

Benefits of DTP Benefits for the Key Stakeholders:

Stakeholder group Current arrangements Future arrangements
Schools – census related activities School administrators operate MIS, entering details about pupils (registration, attendance, achievement, timetable/curriculum, etc. ), workforce, finance, etc. as changes take place and taking into account the competing pressures on their time. Periodically, the school is required to provide census data via COLLECT. The submission of information via COLLECT takes place as follows: Prior to the scheduled collection significant effort is put into ensuring that data relevant to the collection is accurate. This might for example require draft output to be prepared and then reviewed by senior school staff, with any anomalous data being identified and corrected. When ready, data is exported from the MIS and then uploaded to the COLLECT portal. It is then processed (is this immediate or overnight?) and validation errors are identified in reports provided on the portal. The school updates the data and repeats the process until no further errors and submits the data in accordance with the timescale for the collection. School administrators operate MIS, entering details about pupils, workforce, finance, in the same way as at present. There is no requirement for enhanced skills or significant additional training. Information is sent either as it changes or periodically without further manual input.   The emphasis is now on maintaining the quality of data as part of day to day activities, rather than in periodic bursts of activity. Improved data validation within the MIS and wider system will mean that data quality is maintained more easily. This combination of improved local verification of data on entry and validation error reporting within the data exchange system should result in a general improvement in data quality and a reduction on the effort required to maintain data quality within the school. Key uses of data (eg funding decisions or public display of school level data) will still be based on extracts advertised well in advance with time for data checking built into release timetables to protect the integrity of ‘gold standard’ uses.
School – school to school transfer Schools transfer pupil information using the School to School system to support the move of a pupil from one school to another. This requires the export of information from the sending school’s MIS into a standard format and its transmission to the receiving school, with subsequent import into the receiving school’s MIS. Most schools both send and receive information via School to School. The same school staff will undertake the pupil information transfer using DTP without requiring any significant additional training. Possible approaches include: Sending school administrator logs onto the SPDP portal, selects the pupil(s) concerned, identifies which school they will be moving to and when and submit the information. Sending school administrator records which pupils are moving, to which school and when on their own MIS. In either case, the change of ownership will be recorded in SPDP, causing the information to be sent via DE/to become accessible to the receiving school.
Local Authority Currently local authority staff are able to view and update information from schools they are responsible for which has been submitted to COLLECT. Some Local Authorities submit information to the Department on behalf of their schools. LAs also prepare and send specific collections (workforce survey, early years provision). In some cases information is exported from HR systems and reformatted or entered manually into the COLLECT portal. Local Authorities carry out Intelligent Data Checking of information gathered from schools using their local knowledge. Information from LAs suggests that an average LA will have 3-4 FTEs supporting the collection and cleansing of information for submission to DfE. It is anticipated that in the future the LA activities will remain broadly the same, but that the workload will be reduced over time because of improved validation at source of school MIS data.
Awarding Bodies JCQ has recently developed an automated data transfer mechanism based on a point to point solution supporting the exchange of ISB compliant data. This is used by schools to register pupils for exams and to receive results. There is no automated data transfer mechanism for non-JCQ awarding organisations. It has not yet been determined how SPDP will receive JCQ data. The move to the use of DE will mean that all exam related examination within school systems can be fed into DfE via their MIS, reducing the effort for school staff, particularly where they deal with non JCQ awarding bodies. There will be no need for organisational or staff changes within JCQ awarding organisations. Non-JCQ awarding organisations will be able to make use of DE to move data between themselves, schools, and SPDP, potentially reducing any manual workloads.

Benefits for External Stakeholders:

Stakeholder group Current arrangements Future arrangements
Educational Funding Agency Currently EFA uses information collected via the school census process (i.e. using COLLECT) for school funding, with the information being augmented with their own collections and services provided by the Skills Funding Agency.Funding allocations for schools are based on pupil numbers and other information at one point in the year. In the future EFA will have access to accurate and timely information and will have the opportunity to base funding decisions on average numbers of pupils rather than the number registered on a particular day.There will be no impact on the skills required within EFA or the number of staff.Their consumption of SFA sourced data is unaffected. Their consumption of data captured by other methods is less well understood. SFA will be able to choose between existing arrangements, using SFA tools or moving to the new exchange to meet their needs. Further senior discussions are needed to inform the extent to which SFA can gain benefits from DE, given the scope set out in this blueprint.
National College for Teaching & Leadership Currently file transfer processes are used for importing details of new teacher registrations from teacher training providers. Use DE to obtain teacher registration information when possible, or else manual transfer facility. This should be a technical change with minimal organisational impact.
Standards and Testing Agency The STA uses pupil level data initially collected using COLLECT but then schools have to provide information about which pupils sit which SAT and their attainment levels. Up to date information about pupils will be available to STA with no additional activity required from school staff. This should result in a reduction of effort required both within schools and within the STA.STA will need to create a product catalogue to receive ISB qualifications

Sector-Wide Benefits:

Who What the benefits are Delivered by
Children Quicker decision making, for example relating to free school meals (FSM). DE
Improved quality and speed of access to information (e.g. statement details) when a child crosses Local Authority boundaries DE, SPDP
Quicker identification of those going off role – supporting troubled families initiative DE, SPDP
Interaction with school nurses could reduce the risk of ‘falling between the gap’ of immunisation and child measurement programmes when moving schools. DE, SPDP
Parents / public Better access to published school performance data allowing parents to compare schools from a single authoritative source SPDP
Compare own child’s performance with national norms SPDP
Improved public perception of the data DfE publishes DE, SPDP
Schools Increased ability to obtain unpublished but shareable data about their child DE, SPDP
Greater volumes of useful data available for schools to benchmark themselves and for inspection purposes DE, SPDP
Reduced collection burden from LAs and DfE DE
Greater recognition / consistency with DfE ‘published’ data DE, SPDP
Input once, use many times. No need to send same data individually to multiple consumers (e.g. LA, DfE, Ofsted) DE
Common Transfer File always available, not relying on departing school DE, SPDP
Allow easy transfer of information between different school systems DE
Support greater use of data in schools and the research community SPDP
Enabling schools and LAs to drive the benchmarking and analytical agenda DE, SPDP
Local Authorities Speed up HR processes when teachers move to a new school DE,
Reduced administrative costs for things like FSM entitlement DE
Less time chasing schools for data DE
More frequent attendance data and hence better understanding of impact of interventions DE, SPDP
Better information to support capacity knowledge and admissions process DE, SPDP
Enabling schools and LAs to drive the benchmarking and analytical agenda DE, SPDP
Allow existing system compatibility DE
The DfE Cross boundary information immediately available DE, SPDP
More accurate, consistent and timely evidence base to support policy decisions DE, SPDP
Lower cost to obtain and publish data DE, SPDP
More frequent information DE, SPDP
The ability to develop new collections more quickly DE, SPDP
A capability that can more easily be adapted to handle the expansion of the academies programme, new data requirements, and future initiatives DE, SPDP
Support greater use of data in schools and the research community DE, SPDP
Increase internal monitoring DE, SPDP
Enabling schools, LAs and other ‘middle tier’ organisations such as academy groups to drive the benchmarking and analytical agenda DE, SPDP
Putting evidence in the hands of local decision makers, enabling better local decision making DE, SPDP
Improved efficiency by reducing the time spent: processing, analysing and publishing data; responding to information requests SPDP
Reduction in contracted services costs, with less time spent managing contracts and services SPDP
More flexibility in allocation of DfE data analysis staff, with only one data set and less manual processing SPDP
Security management efficiency savings, with less systems to manage SPDP


The scope of DfE collections is:

Alternative Provision (AP) Census;

Children in Need (CIN) Census;

Consistent Financial Reporting (CFR);

Early Years (EY) Census;

Early Years Foundation Stage Profile (EYFSP);

Key Stage 1 (KS1) Assessments;

Pupil Referral Unit (PRU) Census;

School Census (Spring, Summer, Autumn);

School Level Annual School Census (SLASC) – General Hospital;

School Level Annual School Census (SLASC) – Independent Schools;

School Workforce (SWF) Census;

Year 1 Phonics;



Private Fostering; S251 – Budget;

S251 – Outturn;

SCAP – Capacity;

SCAP – Forecast;



CTF (S2S);

School Closure;



Specific Features & Elements to Mention:

The end to end solution will have the capacity to exchange messages containing any data of any format via the Data Exchange Hub.  The Data Exchange Hub will be data agnostic; it will understand how to route the messages (via the message header or wrapper) but will not open the contents of the messages (the message body or payload).

The high level vision for DTP is: “for richer, more accurate data to be available quickly in accessible and usable forms, in order to enable others to drive up the quality of education and services received by children.

Q&A Session on Data Exchange (November 2013)

Who is responsible for training school users of the new system; a) DfE, b) MIS suppliers, c) both?

It’s probably both. LA involvements in articulating new ways of working to schools will help with the culture changes needed. A sector wide solution is likely to require sector wide involvement in implementation.  There will be an element of training and guidance that will need to be provided to users of the system so that they understand the impact of Data Exchange on a particular collection and on their current data flows.  They will also need to understand what will be collected and how frequently so that they can ensure their systems are up to date on a regular basis (maybe more regular than they are used to).  We will also want to ensure current and local good data cleansing activities continue and are strengthened to ensure good, quality and timely data is moved via Data Exchange.

What are the plans for pulling data from the warehouse for use by third parties?

Third parties can currently access data from National Pupil Database – we have existing processes to ensure the security / appropriateness of use and the creation of bespoke extracts.  The warehouse will provide role based access to the data, and if third parties wish to access the atomic data in order to add value and develop tools to support schools / LAs understand their local picture more effectively, that would class as an appropriate use of data, subject to satisfying security levels in terms of how they will locally use and store data.

How will the DfE cope with teachers scepticism?

Early engagement with Schools and LAs has actually been very positive. The high level message has been that the vision is the right one fit for purpose. Although landing it will involve a complex set of challenges, people have so far been supportive of an attempt to reduce the data burdens and improve access to data.  As regards the sceptics, an element of people may not be convinced until they see the tools working of course, but compared to other change programmes the current atmosphere feels very positive.  We will of course continue to engage with Schools, LAs and others as we make our journey through this programme.

What is the security message [being] giving to schools?

The Data Exchange solution will include suitable security controls to ensure that any data sent from a school MIS is protected in accordance with the HMG Security Policy Framework while it is within the data exchange system and will ensure that it is only passed on to end systems that are able to protect it to at least the same standard as the school from which it came. The details of the security architecture and of the control mechanisms required is currently being investigated as part of the completion of the blueprint.

How [will DfE] going to mitigate rumours and urban myths about inappropriate system use to identify children?

This relies on strong communications and significant sector involvement.  Firstly, we have a very strong record of protecting data that we currently hold – there is nothing ‘new’ about the sector moving and holding lots of information about vulnerable children. We have an excellent track record in this regard (although one we would never take for granted).   By ensuring thorough role based access principles, and sector-led governance controls to inform who sees what data when and for what purpose, we can continue to ensure inappropriate data use is a myth not a reality.

Will JCQ be adopting the DE infrastructure when up and running?

There are no plans in place for JCQ or DE to replace each other at present.  JCQ are building an interoperable system to ISB standards between centres and awarding organisations, and the DE will be a much broader application of similar principles.  We are confident that in time efficient movement of data between the two systems will be enabled as a result of both JCQ and the wider DE using a common ISB enterprise data model.

Is ISB going to be mandated in procurement ?

The procurement will be for a hub which is agnostic of the data messages it transports, beyond a certain level of consistency as regards the message header.  However SPDP will only be set up to receive ISB conformant data messages, and so the requirement for messages to flow across the hub in ISB will become essential, but it is not part of a procurement as such. It will be a requirement of systems which want to send messages to DfE via the hub, to be able to communicate ISB messages.

Tagged with: , ,
Posted in Big Data, Data Exchange

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Save A Child’s Life

Register for Child Rescue Alert. Full support from The Eduware Network

Graham on Twitter
Edu. on Twitter
It's a Hit!
%d bloggers like this: