University of London E-Benchmarking

Stage 4: Judging the results

4.1   When the weaving had to stop

The last thread on our sixth and final eMM 'carpet' was tied on 22 November by the eBenchmarking team for the Undergraduate Laws programme. We now have six 'carpets' - grids of squares, 35 rows x 5 columns - in a variety of colours that indicate the performance of each study programme against the 'benchmark processes' set by eMM. Black = Fully adequate, dark blue = Largely adequate, light blue = Partially, and white means things 'just ain't good enough'.

Now we must interpret these results. But first, a moment to draw breath... and put down a few markers of our own.

• Everyone agrees that the process itself, giving programme teams the chance to pause and reflect on their own practice, has been the most useful outcome. The result is a 'programme team view' - and the teams will translate this into improvements through the informal, day-to-day working of the programme whether or not any formal decisions are made.
• The eBenchmarking process would not be as fruitful for the programme team without a 'facilitator' or 'critical friend' to interpret the eMM processes and add a broader perspective.
• However, its value came from being a process led by the programme teams, not a review from outside or above. Its values came from their enthusiasm and hard work. 
• The resulting 'carpets' give a picture of the strengths and weaknesses of the individual programmes, but will not be used for comparison. The six benchmarked programmes range from 40 students to 13,500, from undergraduate to postgraduate, from mixed-delivery to pure eLearning, in subjects from then general to specialised professional development - all too wide for meaningful comparison.
• There are difficulties when considering institutional eLearning policy, management and planning as it is not always easy to identify which institution is being considered in a federal university where colleges and institutes are largely autonomous and responsibility is shared in a variety of different ways.

These were all strong points made at the 19 November meeting of the eBenchmarking Steering Group, which will meet again in January 2008 to finalise its report and recommendations.

In a nutshell, eBenchmarking has proved its value at the detailed level, but we remain sceptical of its ability to offer valid general judgements across an institution as disparate as the federal University of London.

4.2  Is there a future for eBenchmarking in the University of London?

Tuesday 27 November saw a meeting of the core benchmarking team with Jonathan Kydd, Dean of the university's distance-learning division (the External System), its Chief Officer John McConnell and Paul Bacsich, consultant for the HE Academy.

eBenchmarking using eMM has proved its value to six programme teams. How can we use it as a tool for developing strategy?

There is an opportunity here to set standards against which progress can be measured. It is also an opportunity to celebrate eLearning success and diversity.

But it is also clear that eBenchmarking will return benefits only with the willing participation of the programme team. How do we ensure this?

It is important to take the eBenchmarking results and translate these into changes and enhancements - and the resources and funding that can deliver these - for the programmes concerned. The message to all programmes is then: 'If you want to talk about change, this is where you need to start' - with an eBenchmarking exercise.

So is there a future for eBenchmarking in the University of London? It appears the answer is Yes.

4.3   So now we have two tasks...

First we need to wrap up the present eBenchmarking project.

For the HE Academy we've produced a 'single carpet' from the six 'programme carpets'. This will be presented and discussed at the Sheffield eMM Cohort Meeting on 6/7 December. There's the two-page summary to write too. The programme eBenchmarking teams' report are to be collected and collated, and a 'plenary' session of all the teams to be organised. Then there are the report and recommendiations to go forward from the Steering Group.

And the second task is to shape eBenchmarking as a tool for developing distance-eLearning strategy.

This starts on Wednesday 5 December at the meeting with eMM designer Stephen Marshall and colleagues from the Open University: Does the eMM methodology need adjusting for true distance-eLearning, as opposed to eLearning as part of on-campus study programmes?

It's all go.

Stage 3 continued: Here's looking at you, kid!

3.3.8  Time for the mid-term review

eBenchmarking in progress by mid-September: four study programmes. One more started that month: Undergraduate Economics, Management, Finance and Social Sciences at LSE. One more to do in October: MSc International Primary Health Care at UCL. The seventh - the agriculture and development programme from the former Wye College - is in the throes of moving from Imperial College to the School of Oriental and African Studies and is therefore too busy for eBenchmarking in this cycle.

So then there were six. But all doing well. The final 'carpet-laying' session is scheduled for 14 November and we have an outline draft for the final report. And we've had three half-term reviews: by the Steering Group, the core eBenchmarking team and HEAcademy consultant David Kay.

The Steering Group accepted our progress report and agreed Measures for Success (see the previous posting). The core team meeting mapped out the schedule up to completion... and it's tough! We now have just a month before the completed eMM 'carpets' and summary report are to be presented to the HEAcademy consultants and the Dean, as chair of the Steering Group, on 27 November.

We've six eBenchmarking processes to complete; a programme questionnaire to design, distribute, collect responses and collate; six 'carpets' to assess against evidence and summarise; the issue of how we produce a meaningful 'single carpet' resolve; and our final report to write, circulate for review and complete. Panic? Of course we're not!

But it is a tough schedule. At the HEAcademy consultants' interim review, however, David Kay seemed assured we were on track and achieving meaningful results... with a few queries. How, for instance, did our programme teams justify a 'fully adequate' score in the Optimisation column when Delivery and Planning might be only 'partially adequate'?

Well, we have strong eLearning programmes here. Changes have being introduced as a result of last year's experience of eLearning - and that's optimisation, but it is too early in the academic year to judge the results yet. At the final consultants' review it will all come out in the wash!

In the meantime we're already planning for future outcomes from this eBenchmarking process.

How well does the eMM methodology, formed for measuring on-campus eLearning, match the different patterns presented by flexible and distance learning - where study is all at a distance and sometimes wholly on-line through eLearning? We're working with the Open University and the HEAcademy consultants to organise a session at the December eMM cohort meeting to debate this with Stephen Marshall, eMM's creator.

How can the eBenchmarking results from these six study programmes be used to inform changes in the university's use of eLearning? After the flurry of our final report we'll be holding a plenary meeting between all colleagues who contributed, to discuss, compare and debate?

We have built a collaborative website for the eBenchmarking exercise, but haven't had time to launch it and get involvement from the programme teams. How can we use this in the future?

And colleagues of the Wye programmes, forced by pressure of reorganisation to drop out of the current exercise, are keen to be inclkuded in a 'phase 2'... Watch this space!

Stage 3 continued: Measures of Success

3.3.7  Agreed Measures of Success

The second meeting of the UoL eBenchmarking Project Steering Group agreed a set of 13 Meassures of Success for eLearning within the university, based on HE sector standards published in the HEFCE eLearning Strategy but adapted to reflect the operating context of the External System within the University. The document agreed by the Steering Group is attached here:

Download ebenchmarking_measures_of_success.pdf

Stage 3 continued: Motoring...

3.3.5  Undergraduate Laws reaches halfway mark

After second and third meetings, the eBenchmarking team for the Undergraduate Laws programme are now at the halfway point, with 17 of eMM's 35 Processes addressed and assessed. They're keeping the 'carpet' under wraps until it's complete though.

The full process is taking longer than we thought, but this mainly because there's so much discussion - and therein lies the real value of the whole exercise.

3.3.6  Third eMM Cohort meeting / Manchester / 14 September 2007

Interesting to discover the variety of different approaches being taken by our eMM colleagues at other universities. Where we at UoL are eBenchmarking by holding working groups with programme directors, academics and administrators, then looking to documentary evidence to back this up, others are starting with the documents, then following up and filling gaps through interviews.

There's some agreement on our 'Not applicable' issue. Some eMM practices are clearly less relevant to eLearning for 'pure' distance-learning, particularly for postgraduate programmes, compared to on-campus 'blended' learning. 'Not applicable' can also be a positive assessment: training in on-line learning skills is not provided to our MSc Information Security students because these are pre-requisitie for that level in such a subject. 'Not applicable' is thus a positive indicator for the programme.

So one issue for the cohort is how can we compare distance-learning providers' use of eLearning (UoL and Open University) with that of on-campus universities? Colleagues from Sheffield University are willing to explore this avenue - it's an item for our agendas when the pace of eBenchmarking practice starts to ease...

Stage 3 continued: Motoring...

3.3.4  eBenchmarking for four programmes now under way

It's the eve of the Manchester cohort meeting, and the eBenchmarking of four University of London distance-learning study programmes is now under way, with three yet to start.

• MSc Information Security (Royal Holloway) is the furthest advanced, with 30 of the 35 Processes completed in two meetings.
• The Royal Veterinary College's MSc programmes team have met once, but eBenchmarking is now on the back burner, with all attention going to enrolment for the new academic year.
• The MSc Applied Educational Leadership and Management team (Institute of Education) kicked off on 23 August, and their second eBenchmarking session is due on Monday 17 September.
• The Undergraduate Laws programme, meanwhile, fielded seven people for their first session on 30 August, the largest team so far.

And everyone agrees: the most valuable outcome from these sessions is that we're being made to reflect on our own practice. OK, this is something we do all the time... but here and there, not across-the-board. That's why each time a new programme starts eBenchmarking it's taking a couple of hours to get through the first three Processes; it brings with it real discussion about how we're using eLearning.

We're also noticing differences in perspective between New Zealand and London: in discussion it becomes clear that some eMM practices are focused at undergraduate level, others towards on-campus provision. From the perspective of our Masters programmes, with students spread around the world, these practices are being flagged as 'Not applicable'.

But that doesn't matter. Each way we're getting interesting insights. Are others in the eMM cohort  finding the same? We'll find out in Manchester...

Stage 3 continued: On the Road

3.2  Completing the Briefings

We've now run three follow-up half-day briefings for study programme colleagues who were unable to come to the Participants' Briefing on 4 July. Two are on Monday 23 July - for the MA Applied Educational Leadership and Management (Institute of Education) and the BSc programmes in Economics, Management, Finance and Social Sciences (London School of Economics) - and the third on Monday 6 August, for Undergraduate Laws (run by a consortium of colleges).

The news is good. Despite all our efforts they all want to go ahead!

3.3  And we're off!

3.3.1  MSc Information Security

Our first eBenchmarking exercise starts on Thursday 19 July with a session at Royal Holloway for the MSc Information Security programme. With seven people round the table - two academic and two administrative colleagues from the programme, with three members of the project team - we complete just three of the 35 eMM Processes in 90 minutes.

But it's a good start, with a productive approach to the eMM methodology. We're already identifying areas where the use of eLearning specifically for distance education shows strengths, and others where the questions are not relevant. eMM is raisng some interesting perspectives.

All agree that we're resolving issues which will speed things up when we address the remaining Processes. Nonetheless it looks like eBenchmarking for a single programme is likely to be at least a full-day session. This will be on Thursday 23 August.

3.3.2  Royal Veterinary College MSc programme

The first eBenchmarking session for RVC is on Friday 27 July with a team of four people - one academic and one administrative colleage from the programme, and two project team members - and we're outside London surrounded by the fields of Potters Bar.

Again just three eMM Processes are tackled, but again the results are positive and useful progress is being made. But the next session cannot be until Tuesday 23 October. Timetabling is an issue because people are not available: the RVC programme team is small and time is taken up with leave in August, followed by programme starts in September.

3.3.3  Collaboration with the Open University

Monday 13 August sees a meeting with eMM project colleagues from the Open University to compare progress so far and ways in which we can collaborate. The OU is known for its strong student focus in the UK while UoL External Programmes have students worldwide but few here in the UK. So the eBenchmarking of our different approaches to increasing the use of eLearning should yield interesting alternative perspectives - and provide a useful contrast with results from on-campus universities.

One issue discussed is the use of Stephen Marshall's eMM spreadsheet when working in interview situations. It's been shown to have a few 'glitches'. We're working on it...

Stage 3: Briefings and Assessments

3.1   First Schedule for programme briefings and assessments

Following the cohort meeting in Bangor and endorsement by our steering group, we now have an initial schedule. This includes briefings for members of programme teams unable to attend the general participants' briefing on 4 July, and the first assessment meetings to start applying the eBenchmarking methodology to specific programmes. This schedule will be extended as the summer progresses:

Thursday 19 July: MSc Information Security (Royal Holloway) - first assessment meeting.

Monday 23 July: MA Applied Educational Leadership and Management (Institute of Education) - briefing

Monday 23 July: BSc programmes in Economics, Management, Finance and Social Sciences (London School of Economics) - briefing

Friday 27 July: Masters and CPD programmes in Veterinary Sciences (Royal Veterinary College) - first assessment meeting.

Thursday 23 August: MSc Information Security (Royal Holloway) - second assessment meeting.

Meetings for other programmes will be arranged in August. Watch this space.

Stage 2: Defining eBenchmarking for the UoL

2.1   Progress briefing / London / 12 June

We've now had a range of discussions with colleagues - Magdalena Jara from the Institute of Education, who too part in an earlier eBenchmarking exercise; the JISC Regional Support Centre for London, which is based at the University's computer centre (ULCC); programme directors and others.

From these come a first draft Project Overview document - a 'briefing' pack for potential participants and for the steering committee. This both maps out the proposed process and explains the outcome: a 'carpet' of coloured cells identifying areas of eLearning strength (dark blue) and weakness (white) with various shades in between.

An updating meeting with John Dickens, Director of Learning and Development, brings several issues to the fore. The initial proposal is to run initial information gathering ('carpet laying') through intensive workshops with several programme teams present - will this spark debate or bring distracting comparisons between programmes? How do we handle evidence, and how much of it will we need?

Calendars are a problem too. Our HE Academy-provided consultants (Paul Bacsich and David Kay) are keen to speak to the Dean of the External System, Jonathan Kydd, before the second eMM cohort meeting on 11 July, but the Dean is globe-trotting, visiting somjke of the University's collaborating teaching institutions in the Far East and the Caribbean.

And our steering committee hasn't met yet...

2.2   Meeting with the HEA consultants / London / 21 June

The Project Overview document has now reached its third draft. With advaice from the consultants, ideas about eBenchmarking for the university's external study programmes are falling into place. Key points:

• This will be a self-evaluation process by programme teams, not an external audit.
• Initially we aim to eBenchmark 4-6 programmes, with a view to refining the process for use by all programme teams.
• Primary outcome will be a reflective process for programme teams, allowing them to identify areas for improvement in eLearning provision.
• Objectivity will be achieved through a three-stage process: the 'first weave' of the carpet done through a workshop with the programme team; moderation by checking of evidence by the eB project team; then a 'second weave' agreed by a joint meeting.

But we're still uncertain about whether there will be value in 'aggregating' the 'carpets' from our very disparate programmes. At one end there are big traditional distance-learning programmes with 9,000-plus undergraduates and little or no eLearning content; at the other small postgraduate programmes delivered entirely on-line. Won't aggregation reduce these to a meaningless mid-point?

2.3   Participants' Briefing / London / 4 July

Seven study programmes are now on board for UoL eBenchmarking:

• Royal Veterinary College MSc programme
• International Primary Health Care (UCL)
• Undergraduate Laws (consortium of colleges)
• Applied Educational Leadership and Management (Institute of Education)
• Information Security (Royal Holloway)
• Economics, Management, Finance and Sociel Sciences (LSE)
• Agriculture and Development (Wye)

The briefing included members of programme teams plus representatives from the university library and the External System's Performance and Quality directorate. Here are slides from the two presentations:

Download project_briefing_brian_4jul07.ppt

Download project_briefing_liz_4jul07.ppt 

The response is encouraging. Seven programmes is good: this should give critical mass for eBenchmarking all programmes. But will we have sufficient resource? The project team is now four:

• Brian Sayer, Distance Learning Advisor
• Liz Kemp, Planning and Development Manager (convenor)
• Peter Marsden, Strategic Information Ssystems Manager
• Stephanie Wilson, Head of Quality

2.4   Second eMM Cohort meeting / Bangor / 11 July

Seven universities in the eMM cohort gathered for the most useful exchange so far. Guess we're all learning fast! We are:

• University of Leeds
• University of Sheffield
• University of Southampton
• University of Wales, Aberystwyth
• University of Wales, Bangor
• The Open University
• University of London

A conference call from Stephen Marshall in New Zealand answered one of key questions: don't amalgamate the separate 'carpets' where programmes are so disparate. Amalgamation is only valid where deviation across a process is minimal and a 'standard deviation' can be defined. Comparability will give a better guide.

UoL received a public rebuke from the HEA consultants for not having a Blog. Pah! They were impressed, however, by the level of buy-in by the UoL External programme teams. Which matters most, we wonder.

Carpets to be woven and complete for the fourth cohort meeting on 6 December.

2.5   The UoL eBenchmarking Steering Group / London /16 July

Professor Jonathon Kydd chaired the first meeting of the steering group. Papers are attached:

Download steering_group_1_amalgamated.doc

During a focused discussion attention centred on on two key points:

• That this is a formative exercise. The self-evaluation of programmes against a standard framework (eMM) will establish a set of benchmarks for the future.
• That HEFCE's eLearning Measures for Success are acceptable but need to be extended for the External System, expanding those areas specifically relevant to eLearning for distance students. This will be done for the group's next meeting in September.

With minor amendments, the Steering Group endorsed the UoL eLearning Benchmarking Project Plan. Here is the final version:

Download benchmarking_elearning_project_plan_17jul07_ver_5_0.doc

The first live eBenchmarking session is in two days' time...

Stage 1: Preparing the ground

1.1:   eBenchmarking Town Meeting / York / 25 January 2007

The initial impetus for involvement in the eBenchmarking exercise came from the University's Vice-Chancellor, Sir Graeme Davies. As a result, the University's Distance Learning Advisor, Brian Sayer, and Stretegic Information Systems Manager, Peter Marsden, attended the Town Meeting at the HE Academy in York on 25 January.

Outcomes from this meeting were:

• A decision to submit an Expression of Interest to take part in Phase 2 eBenchmarking.
• The selection of eMM as our first-choice benchmarking methodology.
• Contact with colleagues from the Open University, from which came mutual interest in collaboration around the eBenchmarking exercise (assuming both institutions were successful in their applications for Phase 2).

The following Expression of Interest was then submitted to the HE Academy:

Download benchmarking_exercise_2007.pdf

Acceptance was received on 9 February.

1.2:   Setting up the project team / 28 February

First meeting of UoL's internal team - Brian Sayer, Peter Marsden and Planning and Development Manager Liz Kemp - to survey the ground. This produced an initial draft outline for a three-tier approach:

• a three-person team to manage the eBenchmarking exercise,
• a wider working group with representatives from UoL colleges and study programme administration teams, among others, to undertake the eBenchmarking itself,
• a Steering Committee of senior managers to oversee the project.

1.3:  Exploratory meeting with Open University / Milton Keynes / 15 March

The Open University (OU) too was accepted for Phase 2 using eMM, so this was an initial exploratory meeting to consider where our two institutions might have common ground in conducting the eBenchmarking exercise. In particular, this is an opportunity to compare the OU and UoL approaches to eLearning, and to compare these with non-distance-learning models.

1.4:  Phase 2 Kick-off meeting / Bristol / 10 May

An outline of experience from the Pilot and Phase 1, plus introductions to the consultants for each methodology (eMM and OBHE) and the Evaluation and Dissemination Support Team from GlenAffric. Peter Marsden attended for UoL. The team for the OU is Nigel Pigott and Nigel Tonkin.

1.5:  eMM Workshop / Bristol / 11 May

Baptism of fire for all three members of the UoL project team, grappling for the first time with the complexities of the eMM methodology.

One immediate conclusion is that we need a 'lite' version of some kind. The full extent of the methodology, as previously applied by the University of Manchester at an institutional level, is going to be beyond our limited resources - not just the part-time involvement of the three project team members (we've all got day jobs), but the level of commitment required from academic programme teams at multiple UoL colleges.

On the other hand, we need to ensure enagagement with the 35 eMM processes - and be prepared to extend these if our distance-learning context requires this. We decide to select a group of study programmes to benchmark, but which ones? And how many?

1.6:  Second eMM 'cohort' meeting /Milton Keynes / 23-24 May

Getting to grips with the eMM methodology through hands-on exercises using a case studies from the 'University of the South Pole'. This was supported by contributions from eMM creator Stephen Marshall via conference call from New Zealand. Brian and Liz attended for UoL on Day 1, joined by Peter for Day 2.

Now we're ready for Stage 2: Defining eBenchmarking for the UoL

Introducing the University of London and eBenchmarking

Welcome to the weblog of the University of London External System eBenchmarking team, part of the UK higher-education eBenchmarking project being run by the HE Academy and JISC.

The University of London External System (UoL for short from now on) has joined Phase 2 of the eBenchmarking project, and will be using the eMM methodology (eLearning Maturity Model) in conjunction with six other universities.

This weblog will be used to track the participation and progress of UoL eBenchmarking. Please address any queries and comments to Peter Marsden at pandvmarsden@btinternet.com.