Evaluation of an information service

1

Table of Contents
Table of Contents...............................................................................2 1.0Introduction..................................................................................3 Explanation of the information service................................................3 3.0 Evaluation....................................................................................3 3.1 User feedback..............................................................................................3 3.2 Usage data...................................................................................................4 3.1 Survey responses.........................................................................................4 Conclusion.........................................................................................5 Bibliography......................................................................................6 Appendix...........................................................................................6 Sample Knowledge Bulletin email:.....................................................................6

2

1.0 Introduction
This report will explain the evaluation process for an information service. The Information service to be considered is the fortnightly Knowledge Bulletin email newsletter, as sent internally to all members of the professional staff of Davis Langdon, a construction consultancy firm. The Knowledge Bulletin will be evaluated for its utility and take-up among its target audience. This report will focus on the proposed methodology of this evaluation exercise.

Explanation of the information service
The Knowledge Bulletin is an internal current awareness publication that is sent via email to all professional members of staff within Davis Langdon. It is sent using mailchimp.com, an email marketing service. The Knowledge Bulletin is comprised of summaries of articles in technical publications and new library items, and the email contains links to content that is available online. If the document is not available online, it can be provided from the library in either hard copy or electronically by request. Historically, this current awareness publication was circulated as an HTML email, but written in a very basic style. Specifically, it was written in Microsoft Word, where style and formatting was applied and links to documents on the internet were included, and then copied and pasted into a new email message. In the winter 2008/9, the method of distribution was changed to include a web-based email marketing service called MailChimp. This service enables usage tracking, which provides a built-in method of feedback.

3.0 Evaluation
There are three methods through which the library will be able to receive feedback for this service: Unsolicited user feedback; usage data from MailChimp; and a survey sent to readers with the Knowledge Bulletin. The evaluation methods recommended in this report are primarily qualitativerather than quantitative measurements. As long as the Knowledge Bulletin is used by a certain portion of the professional staff of Davis Langdon, it will have served its purpose. It will never be read on a weekly basis by all members of staff, although the quantitative measurements offered by MailChimp’s online evaluation tools will help to measure if the improvements introduced in response to these efforts have been valuable.

3.1 User feedback
The most direct form of feedback on the effectiveness of the Knowledge

3

Bulletin as an information service is that of direct user feedback in the form of emailed replies requesting further information about one of the articles featured. Before the move to MailChimp, this was the only regular method to evaluate the service and to be sure that members of staff were reading and using the Knowledge Bulletin. Keeping a record of the number of people who reply to the Knowledge Bulletin and request material, as well as the number of people who reply with other feedback, provides a very unrepresentative survey of the readers of the Knowledge Bulletin. By definition, the only people who will reply in this manner are those who not only read the Knowledge Bulletin, but those who find it useful enough to spend time to contact the library. While most of the feedback received in this manner is very basic (ie, ‘please send me a copy of …’), users do occasionally include further feedback, such as when explaining that they did not understand some feature of formatting or style. Because this feedback almost exclusively arrives by email, there is no reason to record this feedback in an ongoing fashion—as emails, they are automatically backed up until such time as they are needed for the purpose of an evaluation exercise.

3.2 Usage data
Part of the justification for the move to distribution of the Knowledge Bulletin using MailChimp is the variety of usage data that is included as part of the regular package for distributing a newsletter. Included in this data is information on: • the number of people who opened the email1 • the number of people who clicked on each of the links • how many times each person opened the email • what time of day the email was opened This information is provided by the MailChimp website both in online format and it is possible to export the raw data in .csv format to a spreadsheet programme such as Microsoft Excel. It is available for all past newsletters sent via MailChip, and so as with email replies to the newsletter, it is not necessary to record this data except for the purposes of an evaluation exercise.

3.1 Survey responses
Neither of the previous two methods of evaluation actively solicits recommendations on how to make the Knowledge Bulletin more useful for its intended audience. Despite the note at end of each email that states : Please contact The Library if you have any questions about For a number of technical reasons, this number is not always accurate. However, the other information provided by MailChimp is accurate.
1

4

items in this Knowledge Bulletin. We welcome any feedback. there is no active attempt to solicit feedback on a regular basis. This is probably as it should be, because the nature of the distribution of the Knowledge Bulletin is such that most of its intended audience feel comfortable in giving feedback when necessary. However, occasionally it can be useful to solicit specific feedback in order to improve a service, and that is why it would be useful to conduct a survey in order to help with the evaluation of this service. Probably the best and most effective way of conducting a survey would be to include a link at the very top of the Knowledge Bulletin to a SurveyMonkey.com survey that asks for specific feedback on different portions of the newsletter. As part of a library survey in early 2008, feedback was solicited on a range of library services including the Knowledge Bulletin, but this feedback was simply based on the questions regarding the Knowledge Bulletin in general. Specific questions to be included in a survey evaluating the use of this particular information service could include: • Which sections of the Knowledge Bulletin do you read on a regular basis? (Costs, Technical, Legal… all, none, it depends) • Do you pay attention to the headings / sections, or do you read the Knowledge Bulletin from top to bottom (or as far as you go)? • Would you say that you pay attention when the Knowledge Bulletin arrives in your inbox (a)sometimes; (b)always; (c)never? • What do you primarily use the Knowledge Bulletin for? (a) maintaining professional awareness in your specialist area; (b)keeping current in other areas; (c) Interesting reading; (d) I read it because I have to, but I don’t derive any value from it; (e)I don’t read it because I don’t have the time; (f)I don’t read it because I don’t want to; (g)Other/elaborate. • How could the Knowledge Bulletin be more useful to you? If a survey of this sort were to be included in the Knowledge Bulletin at the top of the email for two successive newsletters, with the addition of a note in the subject line (currently, the convention for the subject line is ‘Knowledge Bulletin XX’ (where XX indicates the issue number), an ‘including survey’ could be amended to the subject) to attract attention,

Conclusion
This report has recommended three methods by which the library will be able to evaluate the use of its fortnightly current awareness publications, the Library Knowledge Bulletin. By regularly evaluating how the product is used, library staff will be able to improve the product to make it more relevant to current and potential consumers.

5

As well as in-depth reviews that make use of all three of these methods, more frequent and informal evaluation exercises can take place using the information provided by MailChimp and by reading all responses received in direct response to the Knowledge Bulletin.

Bibliography
Kuruppu, Pali U. ‘Evaluation of reference services – a review’ The Journal of Academic Librarianship, Vol. 33, No. 3, pp 368–381. ‘CriSTAL checklist for evaluating a user study’ shef.ac.uk/scharr/elib/use.htm. Accessed 27 September, 2009. mailchimp.com http://www.mailchimp.com/articles/how_to_send_surveys_via_html_email/ Word count: 0

Appendix
Sample Knowledge Bulletin email:
KB 87 29 July, 2009

Sections
Costs International Markets Markets - Sectors Regional Technical Sustainability Legal APC & CPD CIS

Costs
Presents the installation and life-cycle costs for commercial biomass heating systems. Systems typically have an expected service life of 15-25 years, and can be classified based on the type of grate that controls fuel combustion. BLP Insurance; Building, 17.07.09, p.75 Examines the progress in retrofitting the UK's existing housing stock in order to meet sustainability targets. Presents a value carbon analysis of improvements in a case study, including CO2 savings and cumulative costs and savings for improvements. Building Sustainable Design, August 2009, pp35-37 ^Back to top

6

International
Studies the costs and benefits of sustainable healthcare facilities in Australia. Notes that on average, hospitals require twice as much energy and six times as much water per sq/m as an office building. Davis Langdon; Health Sector Joins Green Ratings ERA, Research Report, 2009 (PDF download) Examines the contribution to jobs that retrogreening offices in Australia will have, and the part that this will play in an economic recovery for the country. 9% of the country's labour force is involved in the construction sector. Davis Langdon; Retrogreening Offices in Australia, Research Report, 2009 (PDF download) Considers the use of PFI frameworks within the UK and abroad. Lists top PFI legal firms, financial advisors, lenders and construction firms. Also lists the top 10 PFI nations by value of work (led by Brazil, with the UK in 5th place) and by number of deals (led by Spain, with the UK in 3rd place). Construction News, 16.07.09, pp16-17 Costs for sewer, water and drain pipes in selected cities in the US and Canada. Also includes cost and materials indices for major North American cities. Engineering News Record, 13.07.09, pp51-52 Costs for asphalt, cement, aggregate, concrete, brick, block and lime in selected cities in the US and Canada. Also includes cost and materials indices for major North American cities. Engineering News Record, 06.07.09, pp97-98 ^Back to top

Markets
Market forecast. Compares the current recession with the previous three by presenting historical figures for tender prices and for new-build output. Average tender prices are now 10% lower than a year ago. Davis Langdon; Building, 24.07.09, pp51-52

7

The Tracker. Notes that the activity indicator index has risen to 43 (the highest level since 2008) but still represents a slowdown in the rate of activity. Evaluates materials costs, along with a regional evaluation. East Anglia is the only UK region to see a rise in construction activity. Experian; Building, 17.07.09, pp62-63 Building intelligence. Explains that while the infrastructure and public sector markets look promising, all other sectors have seen declines in outputs both year-on-year and from the previous quarter. Total construction output has dropped 16.5% from a year ago. Experian; Building, 17.07.09, pp64-65 Business barometer. Lists the top contractors, architects, clients and QS firms in the year to July 2009. Kier leads the monthly work won in June, and Partnerships for Schools is the top client. Davis Langdon is the 3rd QS firm, with 116 projects worth £1,333m. Building, 17.07.09, pp18-19 Examines the concrete industry. While decreased demand has led to a 33% decrease in orders for ready mixed concrete in the first 3 months of 2009 compared to 2008, the industry hopes that recovery will be aided in part due to gains in market share as concrete is repositioned as a sustainable commodity. Contract Journal, 15.07.09, pp14-18 The latest Federation of Master Builders State of Trade Survey (Q2 2009) records a general slowdown in the rate of decline, but negative trading conditions remain throughout much of the industry. State of Trade Survey, Federation of Master Builders, Q2 2009 ^Back to top

Markets - Sectors
Considers the levels of work in the social housing sector. Lists the top contractors in the sector, as well as the 30 biggest housing schemes in the UK.

8

Notes the levels of starts and completions in England, Wales, Scotland and Northern Ireland. Construction News, 16.07.09, pp24-26 Examines the status of construction work by supermarkets. Clients (led by Tesco) have cut fees (by up to 40% so far this year) and renegotiated contracts to stretch payment periods. The sector still accounts for £2.5bn in work. Building, 17.07.09, pp24-26 Lists the top 10 clients and contractors in the hotel and leisure sector. Also shows the value of regional planning approvals and examines the bright spots in the sector throughout the UK. Construction News, 16.07.09, pp22-23 Focus on healthcare property. Explains that the property market for redundant hospital sites is not strong, and examines the market for healthcare development. Estates Gazette, 11.07.09, pp63-69 ^Back to top

Regional
Hampshire, Dorset and Wiltshire. Property Week, 17.07.09 Scotland. Lists the top clients and contractors, number and value of approvals. Construction News, 16.07.09, pp20-21 Surrey, Sussex and Kent. Estates Gazette, 11.07.09, pp71-79 Central London. Property Week, 24.07.09, pp53-60 Focus on architecture in Birmingham, including the Birmingham Central Library and the Big City Plan. BD Magazine, July 2009 Staffordshire and Shropshire. Estates Gazette, 18.07.09, pp67-75 ^Back to top

Technical
Considers the use of the New Rules of Measurement and how it relates to best

9

practice and cost planning procedures. Assesses the NRM against a number of best practice metrics. Davis Langdon; RICS Construction Journal, July-August 2009, pp26-28 Members of the RICS can view this article online at www.rics.org. This month's RICS Construction Journal has a focus on the New Rules of Measurement. If you have any trouble accessing this resource, please do not hesitate to contact The Library. Highlights the main benefits of the RICS New Rules of Measurement (NRM) and considers the approach to risk and budget setting (cost limits) adopted by the rules. Details the benefits in areas such as training and education, inflation and preliminaries. RICS Construction Journal, July-August 2009, pp8-10 Members of the RICS can view this article online at www.rics.org. This month's RICS Construction Journal has a focus on the New Rules of Measurement. If you have any trouble accessing this resource, please do not hesitate to contact The Library. Explains the alterations that will be taking place at the King's Cross train station as it is refurbished. The £400m renovation programme, which includes removing the green canopy and adding a platform for the East Coast Mainline. New Civil Engineer, 09-16.07.09, pp22-24 Article considering the environmental benefits of fan convectors and how they can help Housing Associations meet the Decent Homes Standard. Housing Association Building and Maintenance, July 2009, pp19-20 ^Back to top

Sustainability
Examines the sustainability of precast concrete paving particularly in relation to new legislation that demands the use of sustainable drainage techniques such as concrete block permeable paving. Housing Association Building and Maintenance, July 2009, pp13-14

10

Looks at the effects of the Low Carbon Buildings Programme and the role ground and air source heat pumps will play in future legislation to encourage the uptake of micro-renewable technology. Housing Association Building and Maintenance, July 2009, pp31-32 ^Back to top

Legal
Examines the use of e-tendering as part of the procurement process in the UK. While there are advantages to e-tendering, and some clients such as the Olympic Development Authority favour it, the private sector has yet to fully embrace the method. Builder & Engineer, July 2009, p.8 Explains the use of contracts as protection from anticipated risks. Advises using credit checks in order to initially investigate clients, and recommends various ways to determine contractual rights if a client fails to pay or fulfil contractual obligations. Contract Journal, 15.07.09, p.28 ^Back to top

APC & CPD
Introduces some principles in environmental building design, including tools for designing with daylight. Provides an overview of how to calculate daylight in a building. Building Sustainable Design, August 2009, pp41-42 Examines the principles of variable flow systems. Provides an overview of all aspects, including pumps, valve controls, system design and commissioning. Recommends further reading on the subject. Building Sustainable Design, August 2009, pp45-48 CPD/information services from the NBS Learning Channels are accessible from http://channels.theNBS.com/davislangdon. Sign in using your username and

11

password, or visit the site to sign up. ^Back to top

CIS
BSRIA blue book, 2009 Annual data book presenting tables of technical data, building running costs, new and forthcoming legislation, contact details for key industry organisations, contact details for BSRIA's staff expertise, list of BSRIA members, industry statistics, key performance indicators (KPIs) for 2009, and publications and services. Blue Book, BSRIA, 2009 Performance and service life in the Environmental Profiles Methodology and Green Guide to Specification Explains the principles behind BRE’s approach to service life determination developed for the update of the Environmental Profiles Methodology and The Green Guide to Specification. Information Paper 1/09, BRE, 2009 ^Back to top

12