Tackling ‘Life after Levels’ with technology

I was recently asked by the Federation to which our school belongs to plan a ‘Life after Levels’ assessment system with some colleagues – and wanted to share here how embracing technology has influenced the design and implementation of the system. The account below is focused on the particular problems of assessing history – but hopefully some of the principles are relevant to other subjects.

We quickly identified the conflicting demands being placed on such a system – see the diagram below, with the blue box representing the immediate concerns of teachers, red those in the medium term and for heads of department and green for data managers:

Screen Shot 2016-01-31 at 21.04.44

We have attempted to prioritise the blue and red requirements, which has sometimes been difficult because of the power wielded by data managers in the current climate of high accountability.

However, we hope we’ve reached a reasonable compromise – much of what we’ve decided to create is based on Alex Ford’s excellent guidance, whilst also taking into account that different schools in the federation will (hopefully) want to tailor their curriculum to be coherent and connected across the key stages (as suggested by Michael Fordham).

The system we’ve designed looks like this and is all based in Access on Office 365, all of the data from the 11 schools in the federation will be anonymised and stored centrally to allow comparisons (this is our small attempt to make use of data in the way I mentioned in my last post):

Screen Shot 2016-01-31 at 21.07.43

I will produce future posts going into each component in more detail, but the design is supposed to deal with some of the key problems facing assessment in history:

Problem 1: A ‘best fit’ approach is attempted to ‘level’ work, often divorced of content:

The mark scheme generator uses a core set of marking criteria which can be mixed and matched according to the assessment question posed. They have to be customised with key content dependent upon the nature and scope of the assessment question. When marking, teachers then select the statements from the criteria which best match the work of their student. The relative ‘value’ of these statements is stored behind the scenes.

Problem 2: Residual knowledge is not well planned for (this is linked to the idea of spaced repetition and therefore may be relevant to other subjects also):

In history the choice of topics and concepts in a curriculum key, but this can be signposted by the use of regular fact tests which recap key previous content before linked topics. The idea is that students will become better able to connect key ideas from one period or topic to another. We’ve begun to experiment with Office Mix to make the creation of rich multiple-choice quizzes easy, but it is still in beta and therefore has limited functionality (what would be great is if this could link its analytics into Access on Office 365, but we won’t hold our breath!).

Problem 3: Teachers spend a lot of time writing similar comments on multiple assessments when they could be engaging with the specifics of student answers:

Crowd source the most effective comments into comment banks, teachers can then select the most appropriate from a suggested list based on the statements which they have already selected for the piece of work (or they can generate their own). These can be printed on stickers and the teacher is freed up to annotate the work and set improvement tasks for the students.

Problem 4: Students and teachers are not well equipped to retain detailed information about performance in assessments, which makes it difficult to reinforce and apply feedback in the medium term (especially between years):

The comments from the comment banks can be stored in a student dashboard in Office 365, hopefully along with their fact test scores and, eventually, photos of their work. One of my colleagues is already experimenting with students reviewing their previous work via structured online tasks as homework prior to their next assessment to help them focus on their areas of improvement.

Problem 5: Meaningful comparison between classes and schools is difficult because of the blunt instrument of national curriculum levels:

The richer data generated using similar analytical mark-schemes should allow comparisons between teachers and schools, which may show groups of students performing significantly above the average, potentially provoking classroom research and opportunities for CPD.

Problem 6: At some point data managers want numbers attached to students’ work and teachers are under pressure to show ‘progress’:

As we are moving to a cohort referenced system at GCSE, it seems worth trying to avoid a criterion referenced system at KS3. By gathering together all of the data across 11 schools, we hope to be able to cohort reference students at any reporting point by combining all of the data available about their performance (formal assessments and multiple choice quizzes). We realise that this will require some statistical trickery, but hopefully the output can be in whatever form is required by individual schools, and will be more valid.

I am sure that such a system could not run without the help of technology. Whether it is better than any alternatives I am not sure – any and all feedback welcome.