Session T1A

Towards the Preparation of a Proposal to Measure Intellectual Productivity of an Industrial Engineering Faculty School Case Universidad de los Andes +
Roberto Zarama
Associate Professor, Universidad de los Andes, Head of Industrial Engineering Department, Bogot D.C., rzarama@uniandes.edu.co

Luis Pinzn
Associate Professor, Universidad de los Andes, School of Engineering, Bogot D.C., lpinzon@uniandes.edu.co

Nstor Jimnez
Instructor, Universidad de los Andes, School of Engineering, Bogot D.C., n-jimene@uniandes.edu.co

Sebastin Poensgen
Industrial Engineer, Universidad de los Andes, School of Engineering, Bogot D.C., s-poensg@uniandes.edu.co

Abstract - During recent years there has been an increasing interest in evaluating and classifying universities. University, in this case, is understood as the institution created by the relation between teachers and students, but focused on research. We understand research as the systematic process for the production of new knowledge. This work approaches a methodology to measure the teachers production within an academic unit. The measures proposed can be used to follow-up on the teachers activity, to prepare and align the individual plans with the strategic projects and, in further stages, to assess the university activity. Index Terms Intellectual Productivity Indices, Measure of Intellectual Productivity, Measure of the production of new knowledge. INTRODUCTION This paper aims at a way of measuring the production of university professors. We have been implementing these measures for the last two years. This work has been developed in the context of an increasing interest at international level in evaluating and classifying the universities. An example is the study carried out by the Institute of Higher Education of the Shanghai Jiao Tong University [1]. We identify as a university the institution that results from the relation professor - student. We believe that recognizing the university as an educational institution focused on research continues to be a valid approach, and thus, its work should be measured by its production in this respect. In our opinion, it continues to be

valid - for the university - to understand research as a systematic process for production of new knowledge. At the university, this new knowledge can be of four types: Individually new, socially new, institutionally new, and/or universally new. This role is even more important in this present moment in history when science and technology seem to play an important role in the sustainable development of nations. In this sense, during the last 40 years, the Organization for Economic Co-operation and Development (OECD) has developed a standardized practice to study the research and development activity (R+D)[2]. A result of this study is contained in the "Frascati Family" Manuals [3]. The Frascati Manuals have become one of the most recognized international standards for this type of studies. Public institutions and various agents in charge of managing policies for the development of science, industrial policies and social policies, have followed the general outline of these manuals to develop measuring tools for the ranking and approach of the R&D. In Colombia, we should mention the Instituto Colombiano para el Desarrollo de la Ciencia y la Tecnologa Colciencias that makes use of this manual to evaluate the country's research groups [4]. Within this same context, from the beginning of the nineties, public pressure emerged in the United States requesting information regarding the teaching activity at their universities. The general perception was that the cost of education was high compared to the outcome. For this reason, measure models were developed to communicate the teachers activity. After reaching a certain level of maturity, these models have been captured in different studies.

1-4244-0257-3/06/$20.00 2006 IEEE October 28 31, 2006, San Diego, CA 36th ASEE/IEEE Frontiers in Education Conference T1A-4

Session T1A
In developing this paper, the following studies have been analyzed: 1) Understanding Faculty Productivity, which includes the results of the Joint Commission on Accountability Reporting and the Delaware Report [5], 2) the report on teachers workload by the California State University [6] and 3), and the study by the American Sociological Association [7]. For the purpose of the measures that will be indicated, we would like to suggest a measuring period of at least three years. This is a reasonable time frame for the creation of a product resulting from the research activity [5] [4]. METHODOLOGY Production of knowledge is the benchmark taken to consider the different activities of a professor: Teaching, research, and institutional development. If these are the activities developed by professors in universities, if universities are educational institutions focused on research, and if research is deemed to be a systematic process for the production of new knowledge, then, measuring faculty productivity must take into account the production of knowledge within its three significant activity issues. We consider different types of knowledge depending upon their possible impact in the environment. For the purpose of the methodology, a link is assumed between the production of the different types of knowledge and the faculty members activities according to the Table I.
TABLE I TYPES OF KNOWLEDGE AND FACULTY ACTIVITIES Type of Knowledge Activity Universally New Socially New Individually new Institutionally new Research Social transfer through applied research and specialized teaching Teaching Institutional Development RESEARCH INSTITUTIONAL DEVELOPMENT 5 5 3.5 3 1 1 0.5 1

its professor-members. Equation (1) presents the general index structure:

Activity Indicator i = Cnc *Inc + Cnca * IncA + Cncf If + Cncd * Id Where i corresponds to each of the three activities: Research, Teaching and Institutional Development, Inc is the index for new knowledge, IncA is the index for new knowledge type A, or higher in quality, If is the training index for other people upon whom influence is exerted, Id is the index for information activities, Cnc is the rate associated with production of new knowledge, Cnca is the rate associated with the production of new knowledge type A, or higher in quality, Cncf is the rate associated with training of other people upon whom influence is exerted, Cncd is the rate associated with information activities.

(1)

Every type of index bears a ranking from 0 to 1. The summation of the rates or coefficients for each of the activities is 10. Therefore, the indicator for each of the activities has a ranking between 0 and 10. In each of the activities more weight is given to components considered as more important within each activity. Thus, the construction of the indicators takes the coefficients presented in Table II and gives a higher weight to those activities expected to receive more dedication from the professors in the academic unit.
TABLE II ACTIVITIES ASSESSED IN THE METHODOLOGY Activities TEACHING Cnc 2 Cnca 3 Cncf 3 Cncd 2

We propose the formulation of three indices, one for each activity, where we address the generation of each of the types of knowledge. The origin of these indices has been taken from Colciencias Proposal for measuring the productivity of research groups [4], and has been adapted to the individual and institutional production not only to research, but also in teaching and institutional development. The research and development indicators provide valuable information regarding the ranking and approach of experimental research and development to various sectors and industries, including academic institutions [2]. Although the Colciencias index is designed to assess groups, its application to research production measuring of individual persons does not constitute a problem. This consideration was taken into account in the preparation of the index [4]. In addition, as the function used in the Colciencias index is more or less linear, this allows for an easy calculation of the index for an eventual research group starting from the summation of the indices of

This classification determines, for each of the activities, how each product is taken into consideration in calculating the indices Inc, IncA, If and Id. Each of the indices in each activity reflects the annualized production in each category. The indices for each category, in each activity, were created as in (2).

index wi

Z G z Pz si z =1 K U wi = si 1
w = 1,2,3,4.

G P
z=1 z

KUwi

1
(2)

G P
z j=1

KUwi

>1

Where

1-4244-0257-3/06/$20.00 2006 IEEE October 28 31, 2006, San Diego, CA 36th ASEE/IEEE Frontiers in Education Conference T1A-4

Session T1A
indexiw corresponds to the i activity index within the category w (indiexw1=lnc, indexw2=IncA, indexw3=If, indexw4=Id, for i: 1=Teaching, 2= Research, 3= Institutional Development), Z is the total number of products within the category, K is the period of time over which the measure is carried out (years), Uiw is the threshold for category w in the i category, Gz is the weight for the type of product where z is classified, and Pz is product z weight. Research articles obtained within the scope of Institutional Development activities Regular full-time positions Coordination of research groups or academic programs Participation in Committees Fund raising 2 1 1.5 1 3

The thresholds Uiw for each category and each activity are specified in Table III. This means that the index does not measure productivity, but instead, it measures the annualized production and corrects it by means of a threshold or maximum value in each class of product that can be construed as the expected production that a teacher must have in the respective categories.
TABLE III THRESHOLDS BY ACTIVITIES AND CATEGORIES Thresholds for Activity (Uwi) Category Institutional Teaching Research Development New knowledge 2 2 2 Products Type A 0.5 0.5 0.5 Training 2 3 1 Information Activities 1 1 1

TABLE VII TYPE OF PRODUCT IN TEACHING AND STUDENT TRAINING Type Gz Student loans addressed Participation in postgraduate academic programs or undergraduate professional electives 1 1

TABLE VIII TYPE OF PRODUCT IN RESEARCH AND TRAINING OF NEW RESEARCHES Type Gz Thesis and Graduation Papers Participation in post-graduate academic programs 1 1

1 2 3 4

The weight for product type Gz where z is classified for each of the activities is shown on Table IV to Table XII.
TABLE IV TYPE OF PRODUCT IN REACHING AND NEW KNOWLEDGE Type Gz Articles covering research on course methodologies or 1 innovations Mass circulation books to be used in courses Book chapters on subjects developed during the courses Creation of Virtual Learning Environments (VLEs) and tools based on TICs Design of new courses Class notes published 3 3/5 2 2 1

TABLE IX TYPE OF PRODUCT IN INSTITUTIONAL DEVELOPMENT AND TRAINING OF MEMBERS OF THE INSTITUTION Type Gz Number of people under supervision Training and development of people supervised 1 1

TABLE X TYPE OF PRODUCT IN TEACHING AND INFORMATION ACTIVITIES Type Gz Course documentation Students perception Products informing on academic results 1 1 1

TABLE V TYPE OF PRODUCT IN RESEARCH AND NEW KNOWLEDGE Type Gz Research articles Research Books Book Chapters Patented or registered technological products or processes Technological products or processes which are not regularly patented or protected for industrial secret Standards based on the results of research 1 3 3/5 3 2 1

TABLE XI TYPE OF PRODUCT IN RESEARCH AND INFORMATION ACTIVITIES Type Gz Product associated to technical or qualified consultancy 1 services Products on information activities or popularization of 1 research results Grey literature and other non-certified products 1

TABLE XII TYPE OF PRODUCT IN INSTITUTIONAL DEVELOPMENT AND INFORMATION ACTIVITIES Type Gz Products associated to the development of Institutional 1 Development activities Producs to inform and popularize the results of the research carried out by Institutional Development 1 activities.

TABLE VI TYPE OF PRODUCT IN INSTITUTIONAL DEVELOPMENT AND NEW KNOWLEDGE Type Gz

The weight of each product Pz expects to reflect not only its existence, but also quality, visibility, circulation, and use.

1-4244-0257-3/06/$20.00 2006 IEEE October 28 31, 2006, San Diego, CA 36th ASEE/IEEE Frontiers in Education Conference T1A-4

Session T1A
Taking the above into consideration, the weights of each product Pz in the research activity, for example, are calculated as follows [4].
Rz Iez (1 + Icz + Ivcuz ) Nz /(Tz ) Rz Iez Pz = R Ie (1 + Ic + Ivcu ) z z z z R Ie N /(T ) z z z z

If it is a new knowledge product other than standards If it is a product associated with training or qualified advice If it is a standard If it is an information and result popularization product

Technological products or processes not normally patented or protected for industrial secret Standards based on research results

Industrial Prototype Pilot plant Product or process protected for industrial secret Regulation or social, education, environmental or health standard Technical standard

1 1 1 1 1 1 2

(3)

Where Rz is the relative weight of product z, Iez is the indicator of existence for product z, Icz is the indicator of quality for product z, Ivcuz is the indicator for visibility, circulation and use of the product z, Nz is the number of authors of product z that are part of the research group, and Tz is the total number of authors of product z.

The relative weight Rz for products associated with new knowledge in the research scope is shown on Table XIII [4]. Each product category is divided into Types and Sub-types. Gz corresponds to the Type and Rz to the Sub-type of the product.
TABLE XIII RELATIVE WEIGHTS ASSOCIATED WITH NEW KNOWLEDGE IN THE RESEARCH SCOPE Type Sub-type Rz Gz Complete articles Short articles Research articles Review articles Presentation of clinical cases or case reports Research books Chapters of books Author book on research results Chapters in books showing results of research Technological Process Industrial Design Integrated circuit design scheme 0.3 0.3 1 1 1 1 0.6 3 1 0.6 1

The value of the Iez, Icz e Ivcuz indicators are shown on Table 4 (Colciencias, 2004, p. 17). Iez is a binary variable (i.e., 0 if the product does not exist, 1 if it exists) taken as the base to accept a product when calculating the Pz. Icz is a variable that ranges between 0 and 3. Icz is a variable that ranges between 0 and 1. The valuation criteria of indices Icz e Ivcuz are dependant upon product classification. The only new knowledge products with Icz = 3 (very high quality) are those to be considered for the creation of the new knowledge index Type A; this means that these products will be taken into account both in Inc and in IncA. Table XIV to Table XVI show the references to prove the authenticity of the new knowledge products (Iez indicator value 0 or 1), and references values for the Icz e Ivcuz indicators (values on the right side of each indicator) [4].
TABLE XIV Iez INDICATOR Type Iez ISSN, name of magazine, title, authors, volume, number, date, or virtual references in the event of electronic publications. Title, authors, ISBN, publisher, city, year of publication, number of pages of book, number of editions. Title of product or process, number or code of patent or register approved by the Superintendence of Industry and Commerce or equivalent, date of patent or register Title of product or process, number of contract or agreement of group with the applicant or product /process user, date of product

Research articles

Research books Book Chapters 3/5 Patented or registered technological products or processes Technological products or processes that are usually not patented nor protected for industrial secret 3 Standards based on research results

New variety (vegetal or animal) or new breed 1 Patented or registered technological products or processes Information development software Specialised application software Analytical process Instrumental process Industrial process Teaching process Therapeutical process 1 0.5 1 1 1 1 1

Title of regulation or standard, number, date of issuance, competent entity responsible for issuance

TABLE XV ICZ INDICATOR Type Icz

1-4244-0257-3/06/$20.00 2006 IEEE October 28 31, 2006, San Diego, CA 36th ASEE/IEEE Frontiers in Education Conference T1A-4

Session T1A
Research articles Published in magazine A Published in magazine B Published in magazine C1 Certification from the evaluation committee or the publisher informing the research nature of book or specification on cover page or inside the book, indicating that it presents research results by authors Patent of invention or its equivalent in other country. Patent of invention or its equivalent in other country. Register Number or code of the institutional or corporate certificate guaranteeing the existence, quality and application of the non-patentable product or process. 3.0 1.6 0.6

FUTURE WORKS This work is not yet completed. There are some extensions pending. One of them has to do with associating the production with the unit's direct costs in Middaugh activity [5]. Still there is insufficient data to assess the relative weight assigned to the indicators' coefficient and to the relative weights indicated in the tables. On the other hand, an information analysis, aimed at establishing the adequate time window for measure performing, for long periods is required . Likewise, the proposed measures describe only the faculty members activity. The teaching activity should not be seen as an aim, per se, but as a means of creating value for students that are now involved with the said activity. In this order of ideas, it is important to consider teaching as a driver towards improving the results obtained by the students (i.e., the rate of graduations, career placements, post-graduate studies, just to name a few possible measures) and to take into account qualitative factors (effectiveness of teaching method, curricular and institutional changes, among others). Therefore, the proposed indicators must be complemented with measures on the effectiveness attained. REFERENCES
Jiao Tong University, Academic Ranking of World Universities, Shangai University, http://ed.sjtu.edu.cn/ranking.htm, 2005. OECD, The Measurement of Scientific and Technical Activities: R&D Statistics and Output Measurement in the Higher Education Sector (Frascati Manual Supplement), OECD, 1989. OECD, Frascati Manual, OECD, http://www1.oecd.org/publications/e-book/9202081E.PDF, 2002. COLCIENCIAS, ndice para la medicin de grupos de investigacin cientfica, tecnolgica o de innovacin, http://www.escuelaing.edu.co/documentos/convocatoria_nacional.pdf, 2004. Middaugh, Michael , Understanding Faculty Productivity , 2001. The Social and Behavioral Research Institute, California State University, San Marcos, CSU Faculty Report, http://www.calstate.edu/acadres/docs/csu_facwrkldrpt.pdf, 2003. American Sociological Association, 2003.

Research books

3.0

Book Chapters Patented or registered technological products or processes

3.0 2.0 1.5 3.0

Technological products or processes that are usually not patented nor protected for industrial secret

Standards based on research results

Number or code of the institutional or corporate certificate guaranteeing the existence, quality and application 3.0 of the product or process protected by industrial secret. Social, educational, environmental or health regulation or standard: Level and scope of the regulation or 3.0 standard: Protocol, law, decree, ordinance or agreement TABLE XVI IVCUZ INDICATOR Ivcuz Use of the product in a public sector Summary of book in indexed magazine. Book with summary in magazine referred in book database with selection committee. Book with summary in magazine referred in book database without selection committee. Translated Book Use of the product in a public sector 1.0

Type Research articles

0.6 0.3 Research books

0.1 0.4 0.5

Book Chapters

Patented or registered technological products or processes Technological products or processes that are usually not patented nor protected for industrial secret Standards based on research results
1

Royalties agreement (number of 1.0 agreement) Product in marketplace (Commercial 1.0 name of product). Product in marketplace (commercial name of the product originated from the industrial secret) Use of the product in a public sector Use of the product in a public sector 1.0

1.0 1.0

Magazines A are those classified by the COLCIENCIAS homologation and index-linking system as magazines A, plus those index-linked in ISI or Scielo. Magazines B are those classified by the COLCIENCIAS homologation and index-linking system as magazines B, plus those index-linked in two or more index-linking systems, other than ISI or Scielo. Magazines C are those classified by the COLCIENCIAS homologation and index-linking system as magazines C, plus those index-linked in one-only system other than ISI or Scielo (the list of magazine index-linking systems used can be found in the ScienTI website under the section Search for Magazines).

1-4244-0257-3/06/$20.00 2006 IEEE October 28 31, 2006, San Diego, CA 36th ASEE/IEEE Frontiers in Education Conference T1A-4