Quality Assurance Policy

At the heart of the LENS community lies a commitment to the development of scholarship and a professional community of practice in the area of adult literacy, numeracy and ESOL. The quality assurance(QA) of the materials held in LENS therefore is an important issue.  However, the approach adopted in the development of LENS has been for the QA to be in the hands of the contributors themselves, rather than controlled by gatekeepers.  The main thrust, therefore, of the QA policy resides with the selection of contributors.

Quality for whom?

The target audience of LENS for which the learning objects and research assets are being produced and collected are staff involved in post-graduate adult literacy, numeracy and ESOL courses in member universities.

Quality of metadata structure

The design team used two methods to ensure, as far as possible, the quality of the metadata structure.  First, they adopted the IEEE metadata standards for learning objects to the extent that these coincided with the aims and objectives of LENS.  Secondly, a careful review of current literature on the quality of metadata in learning objects repositories formed the basis of the procedures by which LENS' metadata structure was developed.  Experts in both adult LLN and information and data services[1] were involved in the choice of metadata fields and the structure.  The metadata structure was then evaluated by contributors using it to tag learning objects that would eventually form part of the repository.  This process will be reviewed as part of the summative evaluation.

Quality of classifications and keywords

An examination of currently available sets of adult LLN keywords showed that there were no ready-made classification systems appropriate for LENS[2].  This is not surprising, as classification systems are typically intimately bound up with the subject matter and aims of the project for which they were built.  The team therefore developed a set of keywords of their own, based directly on their understanding of the body of knowledge that characterises ALLN and on the learning materials and resources that will be entered into the repository.  In this process, the team took full account of the multi-disciplinary nature of  the field and the theory/practice balance that characterises successful CPD at level 7.

Again, it was recognised that the set of keywords could not be comprehensive and perfect from the start.  The database, therefore, has been designed to allow the addition of new keywords, with the acceptance of a QA team[3].  Exact mechanisms for this still need to be developed, but will ideally involve the automatic distribution of all new keyword suggestions to the team every month for their sign-off.

Quality of content of metadata fields

There are two main quality issues in terms of the content of the metadata field.

The first issue revolves around consistency.  Research suggests that however small or intellectually and professionally cohesive a group of contributors, there is wide variation in the way users interpret and use metadata tags (Downes 2003).  The solution of having a single tagger for the materials as a way around this problem was rejected in favour of developing a very flexible metadata structure, which allows users to use multiple classifications, to create new 'collections' of their own and to link learning objects in relationships of their choosing.  It is anticipated that used this way, LENS will in time contribute to the development of new perspectives in adult LLN. 

The inevitable inconsistencies in the use and interpretation of metadata, which the above solution can only partly alleviate, has implications for the retrieval of learning objects.  How can one easily find learning objects with less than perfect metadata?  The mediation of the members of the LENS community of practice is seen as the way forward here.  The LENS database has been designed so that there is a very tight integration of the database of learning objects and communicative tools (e.g. computer conferencing), allowing members of the LENS community to personally recommend resources to other members of the consortium.

Coverage and completeness of metadata

The second issue concerns the extent to which metadata is entered at all.  Research is starting to show that the entry of metadata is problematic in that people appear reluctant or too busy to do so (Currier and Barton 2003).  To assist taggers, the metadata entry forms have been designed so that it is possible to inherit tags from a previous object, thus minimizing the need for re-entering duplicate data. 

Training and guidelines

A third way of improving the quality and coverage of the metadata entered into LENS is to offer training and guidelines to all taggers.  This training will take the form of online pages, and will address issues such as why metadata is important, how to use the tagging tools provided, and descriptions of metadata fields and their values. 

Quality of learning objects and research assets

The above issues deal with the integrity of the descriptors of the learning objects rather than the content itself.  However, arguably the most important, if not the sole, indicator of quality of a learning objects repository is the quality of the objects which reside in it.  Four mechanisms will be used to assure the quality of these objects.

Selection of contributors

LENS is not intended as an open system which anyone can access and which accepts contributions from all.  The most powerful quality filter employed in LENS is the filtering of those who can contribute. The responsibility lies with the LENS quality management team to determine who will be given access to the repository.  In so doing, LENS is recognising the recruitment, research and educational QA procedures of all the institutions which make up the consortium.  It is expected that changes to materials recommended through the institutional QA procedures will be incorporated into the database.

Scope and selection procedure

To assist contributors in deciding whether an object is appropriate for inclusion in LENS, and whether to accept a new university and its contributions, a Scope and Selection policy has been developed. 

Peer review and feedback

As LENS is intended for a small, cohesive community of practice, it was not felt either necessary or appropriate to develop a formal peer review and feedback system.  However, in order to encourage collaboration and capitalise on the ALLN, educational and research expertise of the members of the team, collaborative review and feedback tools have been integrated into the database. 

Specifically, rather than offering a single field in which to enter feedback, each object has attached to it a link to an online discussion in which users and the author of the object can examine how the object worked in practice, and how it could be improved.  In order to assist reviewers in giving comprehensive and constructive feedback, feedback guidelines have been developed.

System-controlled QA

Although frequency of use of an item in a database is not always a reliable arbiter of quality, the database front-end will allow users to select the most frequently used and most recently contributed objects.  Similarly, on a regular basis, objects which have not been used will be reviewed by the QA team, and archived.

Evaluation

A formal evaluation of LENS should be undertaken during its first year.  All the above procedures should be reviewed, as will the quality of the metadata structure, contents, database structure and learning objects. 

References

Currier, S. and J. Barton (2003). Quality assurance for digital learning respositories: how should metadata be created? ALT-C.          

Downes, S. (2003). One standard for all.  Why we don't want it.  Why we don't need it. Presentation to National Research Council, Jan 17th.