On Target and In Time: UKOLN performance measurement and quality assessment studies

Ann Chapman, UKOLN

During the past few years there has been an increasing move to accountability of service delivery, and the promotion of standards and targets by government. A major focus is that of 'education, education, education', and various projects have been initiated to further this - the National Grid for Learning, the People's Network, and the University for Industry. However, each of these initiatives will fail if not backed up by 'quality, quality, quality'. There needs to be quality of personnel, quality of equipment and quality of resource. A variety of resources are involved in supporting these initiatives but often these will be the electronic databases and catalogues of libraries, archives and museums.

The library and information sector has not been exempt from the expectation of accountability; performance measurement, performance indicators and benchmarking are all now accepted methods of demonstrating value for money and standards of service. For libraries and information services there are a range of service provisions that can be monitored; this includes the provision of data, in the form of bibliographic records, about stock available to service users. Bibliographic records are only one component of a library and information service, but support a variety of services and stock management activity - from acquisitions, through enquiries and inter-library loans, to withdrawal of stock. If records are purchased from outside sources, the purchaser is looking for a source that can supply the largest number of records of acceptable data quality for its stock at the time they are required. Lack of quality will require additional effort in editing. Records not available when required will mean local creation of records, duplicating the effort.

For meaningful performance measurement it is important that the right data is recorded and that appropriate indicators are chosen. Orr [1] proposed that six criteria are needed for a performance indicator: it must have informative content, be reliable, valid, accurate and practical, and allow comparison. Performance measurement currently being developed for library services relies on baskets or bundles of indicators; at present few of these indicators relate to the quality of the catalogue. The IFLA Guidelines [2] for performance measurement in academic libraries groups performance indicators under six headings, one of which is Catalogue Quality. The two indicators are known item search and subject search. The draft ISO standard [3] for library performance indicators (ISO 11620) has twelve indicator groupings, two of which - Providing Documents, and Cataloguing - include indicators relating to catalogues. In addition to these, the CIQM Database Labels scheme could potentially be applied to library catalogues.

Quality assessment and performance measurement of library catalogues and databases has been a primary focus at UKOLN (and its predecessor organisations, the Centre for Catalogue Research and the Centre for Bibliographic Management) for the last twenty years. In 1980, the initial research proposals for the CCR identified four aspects of bibliographic record quality accuracy, consistency, timeliness and functionality. Since then the centre has undertaken a range of work in this area.

Currency of BNB records
The first study was designed to look at the timeliness, or currency as it became known, of bibliographic records created for the BNB files on the British Library database [4] and [5]. As far as we are aware, the survey provides the only publicly available, externally measured data on the performance of a national library. In addition it is a continuous, long term survey; the collection of samples at the point an item was ready for catalogued was begun in 1980 and a second sample at the point an item was authorised for ordering was established in 1988.

Each month samples are sent to UKOLN, where they are matched against the BNB files. If a record is found, it is designated as one of three types of 'hit'. These are CIP records, formerly CIP records and full (i.e. BNB created) records. If not found, it is a 'miss'. This data is used to calculate the hit rate for the BNB file. A number of factors may skew the results for an individual month and the hit rates are therefore based on a conflation of 12 months data. The hit rates provide a performance indicator on the ability of the British Library to provide a bibliographic record for all monographic printed items published or distributed in the UK since 1974. The current figures indicate that records can be found in the BNB files for 85% of sample titles at either cataloguing or ordering stage.

Any title, for which a record is not found initially, is checked against the file six months later. This provides a recheck hit rate. While it is unlikely that libraries will be prepared to wait for up to six months for a record, the recheck hit rate provides an indirect indicator on the functioning of the legal deposit system. Since a record is only created or a CIP record confirmed only when the required copy is deposited, the lack of a full record can indicate non-deposit. However, cataloguing backlogs at the British Library which sometimes occur can delay the appearance of a full or formerly CIP record and this precludes using survey results as a direct indicator. There is however a strong indication that a proportion of items are not deposited.

Currency of records from other sources
The centre invests time and effort into collecting the samples for the BNB Currency survey. The BNB files are now not the only source of records available to libraries and it was decided to use the ordering stage sample to look at the availability of records from a number of sources. Because the sample parameters were designed to monitor the BNB files, they were narrower than the parameters of most of the sources in the comparative study. For instance, the majority of the databases were not restricted to the UK imprint, and several contained records for material other than printed monographs. The study looked at book trade and library services from Bibliographic Data Services, Book Data and Whitaker, the inter-loan resource union catalogues of LASER and Unity, and the co-operative union catalogues of BLCMP (now Talis), CURL, OCLC and SLS (now Innovative).

The survey began in mid 1996, using the BNB currency survey methodology and the ordering stage sample already collected. The hit rates were produced from conflated data from twelve sample months [6]. Once the hit rates were established, there was little variation in results over time. It was therefore decided to close the study in mid 1999; however, it is planned to survey the same sources in two or three years time to see whether there are any changes in the longer term. Overall the survey found that there is good coverage of the UK imprint, and the majority of the databases serve the academic and public library sectors equally well [7]. Frequency of update is only one factor determining performance levels, as evidenced by monthly CD-ROM services having hit rates of over 90% and the quarterly updated database a hit rate of 88%. Specific factors such as the exclusion of specific types of record can, however, lower performance rates.

BLCMP database quality
The currency survey work focuses on only one aspect of record quality - that of timeliness. In 1992, CBM wanted to look at other aspects of record quality and BLCMP was interested in evaluating the quality of records on its database. This provided the opportunity for a collaborative study [8] to find out what proportion of records in the BLCMP union catalogue were edited after creation and the reasons for editing.

Member libraries retrieve records from the union file and attach new holdings data to them. At this point they may also edit the record. System data was used to record the total number of records retrieved and the number of records edited in March, April and May of 1992. Using the figures obtained, the proportion of edited records each week was calculated to provide a quantitative indicator of record quality for 1992; this varied from 17% to 29%, with an overall average for the three months of 23%. Member libraries made the majority of these edits; BLCMP staff made the remaining edits.

The study also looked at the reasons for editing. A randomly selected sample of 1310 edited records was collected each week for three months. For each record, before edit and after edit versions were printed out and analysed. Possible corelations between editing and the organisation creating a record, and between editing and the various formats (monographs, music scores, music recordings, videos, etc.) were examined. Various groups of edits were identified - reversal of main entry and added entry points, all data in a record changed (usually the result of errors in ISBN allocation or use), subtitles added to records, and the editing out of capital letters in the titles of Whitaker created CIP records. Other edits were new fields added, fields deleted, data added or deleted in existing fields, spelling and transliteration amendments, capitals changed, subfield coding changed and field indicators changed. Some fields were more likely to be edited than others. A specific problem was identified with records that had been converted from other systems; in these cases the records tended to be in upper case text throughout and contained minimal data.

In the final part of the study, member libraries were asked to supply a random sample of ten records each that they had edited and to annotate the printout with their reasons for the edits. These annotations revealed a range of views and reasons for editing. There was concern over the use of more than one transliteration scheme and the fact that filing indicators were often incorrect for titles beginning with definite and indefinite articles in Welsh. Certain sources of records were perceived to be of a low standard. One worrying fact to emerge was that some libraries were deleting fields from the records because they did not require them instead of accepting the fuller record and omitting that field from their output profile. Some of the annotations indicated lack of adherence to BLCMP guidelines. The identification of these issues enabled BLCMP to do further investigation and to address the problems.

The study identified that edits were made for accuracy, consistency and functionality. However, these aspects are often not separable. For instance, spelling errors and different transliterations will both result in misfiling of authors in listings, with either minor or major displacements, and in the non-retrieval of items searched for by author or title. Here accuracy and consistency have an impact on functionality. While correction of publication date seems initially to simply be an accuracy issue, it is also a functionality issue as the database allows publication date as a search term. Field 300 is seen as having a functional value in identifying where material is shelved or indicating the possible value of an item to a search (is the searcher happy with a pamphlet, or are they looking for a more substantial text?).

The figures quoted here relate to 1992. This study could be repeated by BLCMP on an occasional basis to monitor the situation. In addition, other databases that allow member libraries to edit records in a union file could use the same methodology to examine the quality of their database.

CLSCP audit
From 1990 the creation of records for the BNB files has been shared under the Copyright Libraries Shared Cataloguing Programme (CLSCP). The programme required the six libraries (the British Library, the National Libraries of Scotland and Wales, the University Libraries of Cambridge and Oxford, and the Library of Trinity College Dublin) to create records to the same standards. It was recognised that some form of assessment was needed to audit record quality and consistency and this was originally undertaken internally. As the libraries did not have the staff resources to continue this, in July 1994 UKOLN was contracted to carry out the assessment, and continues to do so.

UKOLN does not have access to legal deposit copies of titles so no checking of records against physical items can be carried out. A printout is generated with a random sample of records created. Each printout contains records from each of the six libraries, in the same proportion as the records that are contributed to the CLSCP. Thus 100 British Library records are checked plus nine from each of the other libraries, reflecting the 70% : 30% division of record contribution. From July 1994 to December 1997 this printout was generated each month. Over the period since July 1994, the number of errors found has decreased. The Steering Committee decided at the end of 1997 that monthly checks were no longer required and opted for an annual snapshot check, beginning with January 1998.

The two aspects of record quality under scrutiny are accuracy and consistency. Records are checked for consistency between the same information held in more than one field (e.g. dates in 008 and in 260), for accuracy in spelling, for correct use of field tag indicators and specific fields used. Checks are made that all added fields required are present and that series entries are correct and consistent. An analysis of the errors found is sent to the CLSCP Steering Committee and an annotated copy of the record(s) containing error(s) is returned to the Library that created the record(s). The libraries use this feedback from the audit process to identify why specific errors were made, and use it in training and practice review processes.

Self-assessment of library catalogues
In early 1997, Essex County Libraries approached UKOLN with a request for any knowledge or information the centre had on methodologies for assessing the quality of records in automated library catalogues. They were interested in a retrospective exercise to assess the quality of existing records and in an on-going procedure to monitor the quality of new records being added. Current and previous UKOLN work in this area was not directly relevant and searching revealed little in this area. However, details were found of an audit process developed in Tennessee using a statistical methodology called 'attribute sampling' [9] and [10]. This seemed to require more statistical expertise than most libraries could call on, but did incorporate some important points. Essex and UKOLN decided to collaboratively design a simple method which libraries could administer easily. The design stage was completed but no pilot audit has yet been carried out. (For various reasons Essex has not been able to undertake the pilot and another partner library for the pilot has not been found.)

Two samples are required for the audit. The first requires a computer generated printout of a random selection of catalogue records. This sample is first checked against the shelves and the circulation records to find out whether items are missing from stock. If the items are found (either on the shelves or when returned from loan) the catalogue data from the printout is compared with the book. A set of error types is used to enable simple check off procedures to be used. There are three groups of errors - non-matching data (e.g. the classification number on the book is not that on the record), accuracy of data (spelling errors in the record) and omission of data in a record. This first sample is also used to generate a second sample; the 5th item to the left of the printout item is selected. The items in the second sample are then checked against the catalogue to establish that catalogue records are present for them. The audit produces three figures: the proportion of missing items in the sample, the proportion of records with errors and the proportion of items which have no records. The set of records with errors can be further analysed to find out whether specific types of error predominate.

The future
Libraries and information services will increasingly be asked to monitor and publish information on performance. To do this they need to use appropriate indicators that are agreed for their sector. UKOLN's work is intended to benefit the library and information community in practical ways and methodologies developed in research projects are intended to be extendable to other sources and services and as tools for use by others. UKOLN will continue its mix of long-term work and individual studies in this field.

References

  1. Orr, R.H. Measuring the goodness of library services: a general framework for considering quantitative measures. Journal of Documentation, 29, 1973, pp 315-332. Back to text
  2. Te Boekhorst, P. Measuring quality: the IFLA Guidelines for performance measurement in academic libraries. IFLA Journal, 21(4), 1995 pp278-281. Back to text
  3. Carbone, P. The Committee draft of International Standard ISO CD 11620 on Library Performance Indicators. IFLA Journal, 21(4), 1995 pp274-277. Back to text
  4. Chapman, Ann. National library bibliographic record availability: a long term survey Library Resources and Technical Services, Vol.39 (4) Oct. 1995 pp345-357. Back to text
  5. Chapman, Ann. 1994 revisited: a year in the life of the BNBMARC currency survey International Cataloguing and Bibliographic Control, Vol.26 (2) Apr/Jun 1997 pp41-46. Back to text
  6. Chapman, Ann. Bibliographic record provision in the UK Library Mangement, Vol.18 (3&4) 1997 pp112-123. Back to text
  7. Chapman, Ann. Bibliographic record provision in the UK: measuring availability against demand. Bath: UKOLN, 1998 ISBN: 0-9516856-5-1. Back to text
  8. Chapman, Ann. Up to standard? : a study of the records in a shared cataloguing database Journal of Librarianship and Information Science, Vol.26 (4) Dec 1994 pp201-210. Back to text
  9. Kiger, Jack E and Wise, Kenneth. Attribute sampling: a library management tool. College and Research Libraries, 54(6) November 1993 pp537-549. Back to text
  10. Kiger, Jack E. and Wise, Kenneth. Auditing an academic library book collection. Journal of Academic Librarianship, 22(4) July 1996 pp267-272. Back to text

Further details on UKOLN's bibliographic management studies can be found at:
http://www.ukoln.ac.uk/bib-man/



Next issue


© Information Automation Limited and CIQM, 1999
Links checked: 02 January 2000 15:17:19
This page last updated 02 January 2000 15:17:22
Pages maintained by IAL