TECHNOLOGY MANAGEMENT 204 Dr.

Rolando Dayco 1st Semester 20102011

3
Volume

Management of Research and Development

A N E V A L U AT I O N R E P O R T O N T H E

Philippine Research, Education and Government Institution NETwork (PREGINET)

Submitted By:

Alvarez, Mary. Joan Cuevas, Juan Ceasar M. Derecho, Charisse C.
October 2010

Table of Contents
Topics I. II. III. Objectives of the Evaluation Background of the PREGINET Project Evaluation of the PREGINET Project A. Problem Identification and Proposal Stage B. Implementation Stage C. Post-Evaluation Stage Conclusion List of ReferencesAppendices Chapte r 1 2 3 4 5 6 7 8

1
Page No. 1 to 2 3 4 5 6 to 8 9 10 to 12

IV.
V.

Objectives of the Evaluation

T

his evaluation report aims to assess the Philippine Research, Education and Government Institution NETwork (PREGINET) project as a research and development (R&D) project focusing only on the aspects that has direct impact to research and developmentR&D. Specifically, the objectives are to: To Specific, Measurable, Attainable, Relevant and Time-bound • • Ddetermine the concrete research and development (R&D)R&D output/s of the project; To Ddetermine how effective the Assess the PREGINET project, PREGINET, in terms of project phases: Planning, Implementation and Post Implementation

seekssought to accomplish the following objectives: ii . and in terms of R&D • • To Assess the impact of PREGINET in terms of its R&D objectives and output Assess the impact of PREGINET in terms of its contribution to the advancement of network technologiesPprovide recommendations effectively managefor the management of future DOST-funded R&D projects. with available bandwidth on which next-generation network technology and services can be testinged and deploymentdeployed. 2 Background of the PREGINET Project I n 1996-1997. Thehe project. based on its documented research proposal. There became a trend that industrially-advanceially advanced countries have embracinged a knowledge economy and realizing realized the value of setting-up Research and Education Networks (RENs) to foster innovation. ASTI found this e relevant to the Philippines and thus identified the “need” for the Philippines country to have a network that is contiguous and wide in coverage. and demonstration can be conducted. experimentation. and where applications development. The PREGINET project was conceptualized with all these in mind. During those times. Unfortunately. the strategy of the RPWEB RPWeb initiative was for government agencies to connect through commercial ISPs.was managed from the planning and the proposal stage. to the implementation and post-evaluation stages and how each of these stages contributed to the success or failure of the project. they experienced varying levels of service and there was a a lack of sufficient local connectivity between commercial ISPs.

and business iii . facilitate the introduction of next-generation network technology and services by providing better network infrastructure and new services (e. and said network will be used in the development and demonstration of novel applications for science. education.• • • foster partnerships among government. and key government institutions.g. etc. quality of service. health. academe. and the research community all over the country. security. education.) for research. industry.

10 Advanced Science and Technology Institute – Department of Science and Technology (ASTI-DOST) Philippine Council for Advanced Science Research and Development (PCASTRD) and Technology Implementing Agency Monitoring Agency Cooperating Agencies DOST Agencies and Regional Offices. 2007 – December 31. Education. 2000 – May 31 2005 1st Extension: June 1. 2006 – February 28. 2005 – May 31. 2007 Php78. and Government Information Network (PREGINET) June 1. PREGINET Partners Figure TableFigure 1 PREGINET Project Information Evaluation of the PREGINET Project TI .887.P R E G I N E T E V A L U A T I O N R E P O R T 3 The PREGINET project details are summarized in TableFigure 1 below: Program Project Duration Total Cost Project Comprehensive Program to Enhance Technology Enterprises (DOST-COMPETE) Virtual Centre for Information Technology Innovation in Information Technology (VCTI-IT): Development of a High Performance Research and Education Network: The Philippine Research. 2006 2nd Extension: June 1.793. 2007 3rd Extension: March 1.

(January 2002) . Problem IdentificationConcept Development aAnd Proposal Stage. Project Evaluation Guidelines. Post-iEmplementation evaluation Evaluation stagStagee .Use of Funding . 3) can be justified on the grounds the market is failing to efficiently deliver 1 Department of Treasury and Finance Government of Western Australia.Staffing . which requires that the objectives of the project are: (1) are consistent with the agency’s strategic objectives.Risk Management . Implementation Sstage.Product Development C. DevelopmentConcept A Problem IdentificationConcept Development and Proposal Stage I n this section. 2) are are consistent with general Government policy. we view the project in terms of its strategic justification 1. and .Evaluation Team .Evaluation Methods DAP Evaluation ResultsFindings/ Results of DAP Evaluation DAP Recommendations.P R E G I N E T E V A L U A T I O N R E P O R T n order to determine how effective the Phe projectREGINET project was managed. it will be evaluated in terms of thein the context of each of the three project management stages in project management: A.Evaluation Objectives .Research Outputs . Problem Identification and Solution Proposal Project Objectives and Expected Output B.

undiscovered and cutting edge technologies then that will warrant the need for a high-bandwidth network. however. is in line with ASTI’s mandate. . The implementer’s proponent’s rationale goes: in order for us to research and develop advanced networking technologies we need to first spend our budget on a network. Unfortunately. ASTI. the capabilities of commercial internet service providers (ISPs) are bound to catch up and will be sufficient to meet the needs of non-research government firms. A of alternative solutions to the problem were not identified and there was no an assessment of whether the proposed solution is the best option to address the identified problems was not conducted. The problem identification process and the actual identification of the problems identified that are to be addressed by ASTI is are the basic initial weaknesses ofin the project. It is a case where the implementer proponent attempts to do too muchalready has pre-conceived solution. If they had a clear R&D strategy with a project portfolio and strategy for product development of advanced. The second problem . unlike the first one. The project implementerproponent. and Network dedicated to research. the approach to the solution is again weak and should ha’ve been re-evaluated. interconnectivity of government networks.P R E G I N E T E V A L U A T I O N R E P O R T the proposed project outputs. The first problem. This can be done without DOST-ASTI spending their R&D budget to subsidize these government agencies’ internet connection. and 4) take advantage of and has consideration for current and expected future economic conditions. the R&D strategy is also weak as will be described in later sections. Given the project’s initial timeframe of five (5) years. essentially identified the following problems and/or needs that arewere to be solved addressed by the PREGINET project: • • • Interconnectivity of government networks. Advancement of ICT Information and Communication Technologies (ICT) in the Philippines. is out of scope of the institute’s ASTI’s mandate. . which is to “Conduct scientific research and development in the advanced fields of Information and Communications Technology and Microelectronics. which led to the flawed problem identification process and in with inaadequate screening and evaluation.” Interconnectivity on a nationwide scope is the DOTC’sDepartment of Transportation and Communications’ roleand is essentially DOTC’s burden to bear.

sourceforge. The proponent’s rationale is:: in order for us to promote the advancement of ICT in the Philippines. the project was abandonedfizzled.. ASTI could have focused on determining ways to advance ICT in the Philippines by creating a well-balanced project portfolio with clear R&D strategies for the development of advanced. Aan example can be made of the Pawikan Project. and it would also serves the same purpose as one that is administered by ASTIPREGINET. However. we need to first to set-up a dedicated network infrastructure for local research institutions to have access to international research papers. To address the need for a research networkMoreoverFurthermore. a research network run solely by a a private telecommunications company company would hav’ve been cheaper as market competition between internet service providersavailable ISPs lowered the ratesdrives the rates down.2 Pawikan is a network monitoring software that was developed under PREGINET and was ready to be sold to a a local telecomstelecommunications company. testing and demonstration of advanced networking technologies. In fact. What was left was a source code that no one in the organization i was willing to study and the thus. 2 Pawikan Network Monitor. PREGINET services are not as competitivecannot compete in terms of ratesbecause they acquire the internet connection also through these ISPs and even the network reliability of the networkis reportedly problematic. Needless to say. As for researches.P R E G I N E T E V A L U A T I O N R E P O R T Suggested: The second and third problems are in line with ASTI’s mandate. Having a high-bandwidth network is neither a necessary pre-requisite nor the only means to achieving the desired objectives and virtual research communities can be organized even without setting-up the said network infrastructure. however. with the materials that ASTI can access via their partnership with Asian Internet Interconnection Initiatives (AI3)AI3 and Asia-Pacific Advanced Network (APAN)APAN. facilitate knowledge exchanges and facilitateenable the development.net/ . Ddue to lack of proper documentation and turnover of the source code. undiscovered and cutting-edge technologies. What could have been morewas more feasible and efficient to advance ICT in the Philippines is towas to focus their efforts on purely R&Ddeveloping new network technologies projects and taking steps to ensure their commercialization. the knowledge left when the developer resigned. the approach to the identification of the solution is again weak and should have been re-evaluated.. Unfortunately. the R&D strategy is also weak as will be described in later sections. http://pawikan. they could have provided in a web portal where local research agencies institutions can visit to gain access and thereby encourage more participation and sharing of research outputs.

On the surfaceIn the literal sense. they are simply answerable with Yes/No and are neither measurable nor tangible. Thus it is possible that practically anything can happen in the project and the objectives will still be met. the objectives and output are assessed. However. Questions like How. it is possible that anything can happen in the project and the objectives will still be met. Thus. they are simply answerable with Yes/No and are not measurable or tangible by any means.P R E G I N E T E V A L U A T I O N R E P O R T Objectives and Expected Output In this section. the objectives may sound are clear and plausible. We look at how clearly they were defined and how success and failure of these objectives will be useful for the proper evaluation of the project. the objectives may sound clear and plausible. The expected outputs and cannot even be seen or measured. However. To say that Tthe expected outputs are vague would be an understatement. . The project proposal states five main objectives and one generic statement for the expected output (Figure 2). On the surface. Figure 2 PREGINET Project Objectives and Expected Output The project proposal states five main objectives and 1 statement of expected output (see figure 2).

In view of the foregoing. Given that DOST mandated the thrust towards Science and Technology (S&T) development/advancement (?) and wants sought to address the ICT needs of the Philippines. the projectthe intention is to aimed to utilize ICT technologies. implementationsed and use of tilized network technologies in 5 years of operation B The objectives could have been patterned after “SMART” objectives.setting stage. patents granted. Measurable. instead of normal long-distance calls and travel. Commercialize at least one (1) new networking technology per year. . Attainable. The following are a few examples: • • • • Attain X atotaln X amount of savings due to usage of PREGINET services such as video conferencing and voice over internet protocol (VOIPVoIP) in government agencies. The overall goal of . How many or How much should be answered by the expected outputs. and Government Information NETwork (PREGINET)PREGINET. Network network services and R&D on next -generation products to producein a high a high -performance Research and Education networkREN. Implementation Stage T he project implementation framework is summed up by the diagram below (Figure 43)(Appendix 1). Education. Reach an a X number of partners who’ve published research papers in referreed journals. Wwe can say that It is hard to say if this is intentional. the project should have been rejected or have been returned for revision and further re-evaluationed or even rejected. but at this point in the planning process. etc. “SMART”SMART stands for Specific. a mnemonic used in project management during the project objective. and Achieve an X percentage measurable increase in researches published in scientific papers. and . which is the (Philippine Research. Relevant and Timebound.P R E G I N E T E V A L U A T I O N R E P O R T What.

3. Information Technology IT Industry User Communities for Agriculture User Communities for Distance Education User Communities for Networking Technologies User Communities for Disaster Mitigation User Communities for Bioinformatics and Telemedicine Based on the approved project proposal. and equipment. Figure 3 PREGINET Project Conceptual Framework Its The project’s direct beneficiaries are the following: 1. and for the development. education. 2. 5. Aat the end of thise project.P R E G I N E T E V A L U A T I O N R E P O R T PREGINET the project is to establish a nationwide broadband network to interconnect academic and research institutions as well as government agencies in order to strengthen the country's science and technologyS&T capability and further to enable e-governance. 1. these were the expected deliverables are: Operational exchange point for research. 6. . Network testbed for research in and testing of next generation network technologies. 2. and government network interconnection. 4. services..

telemedicine. Developed and demonstrated novel applications for e-commerce. 4.P R E G I N E T E V A L U A T I O N R E P O R T experimentation. 3. The areas for analysis are the following: • • • • • Research Outputs Return of Investment Staffing Risk Management Product Development . Connectivity to research networks outside the country. 5. and. 6. . Analysis on Implementation Stage The analysis of the project’s iImplementation stage focuses on major areas that are crucial to the success or failure of PREGINET as an R&D Research and Education networkDevelopment organizationproject. distance education. distance education.. Competency in network technology integration and network services provisioning.. among others. and other areas. and demonstration of novel applications in the areas of ecommerce. Competency in network equipment testing and integration services. telemedicine.

among others. Some areas of research done during the project are: Network Administration (DNS. ProxyProxy). it was discovered that no new generation technologies wereas developed. it was also discovered that they the project hashad no established mechanism way of concretely concretely monitoring or documenting what new -generation network technologies technology researcheshthe R&D were conducted and facilitated done through the network by their partners (See Appendix 2). etc). AP BioBox. . wireless broadband technologies. However. Moreover. Furthermore. It could have been more cost effective andand the research staff can focus more important endeavors such as on thethe development of new network technologies a better focus of their skills. ATM. it would have been a more cost-efficient strategy if the project did not spend for the setting-up ofoffset-up the network infrastructure and instead also. the project should have been directed at testing these technoloused the services of cheaper and more reliable services of gies in commercial ISPs. FTP. experimentation. but instead. TThese are matured technologies already being used widely abroad and the findings show that no no new radical or incremental research was includedinnovations were produced by the project. (See Appendix 1).g. and IPv6 Implementation. Aamong the functions for the network testbed would be to alloware to enable experimentation and testing of new technologies and equipment (e. However. since these technologies can also be tested using commericialcommercial ISPs. Multimedia over IP (Videoconference. it was merely adaptation adoption of matured technologies made developed abroadreadily available in the market. Videostreaming). telemedicine. Mail. and equipment. manpower cost can also be reduced as deployment and network support will not be required rather than establish a high speed network where they can test it. Virtual Classroom System. Network Management. DSL technologies. and demonstration of novel applications in the areas of ecommerce. Digital Content System (Library Management System. and for the development. distance education.P R E G I N E T E V A L U A T I O N R E P O R T Research Outputs Included in the intended deliverables of this project are the establishment of a: N network testbed for research in and testing of next -generation network technologies. IP over SONET/SDH. services. Web. Digital Library). Voice over IP. In addition. at the assessment of the final status of the project.

P R E G I N E T E V A L U A T I O N R E P O R T If one of PREGINET’s main thrust was increaseincreased R&D research onof new network technologies or other advanced system applications or tools. Nevertheless. Number of patents generated . Due to the unavailability of information on actual breakdown or components of the project budget. Return on Investment Overall. we cannot adequately assess whether the budget spending was cost-efficient. the approved budget for the PREGINET project was P79M (Figure 4). Amount of external research grants obtained and Return on R&D investment. we have established the fact that the project budget could have been spent more efficiently and effectively if it was used for the development of new products that can be commercialized instead of funding the infrastructure set-up. . internet connection and network support to partners. as we have earlier identified the project’s deficiency in problem identification and solutions-matching. Number of innovations commercialized . their quantitative measures of success could have been the following: • • • • • • Number of refereed papers published . Number of innovations developed .

Consequently. the funding for the PREGINET project was a total of P79.60. Tthe PREGINET has not been successful in proving that this project was unable to come up with anything that would constitute agenerate revenues for the government except the minimal service charges they recently introduced. Released. Maintenance and Operation Expenses and Capital Outlay. In fact. PREGINET has been subsidized and included as part of ASTI’s annual budget. However. . Obviously. Return of Investment. A Cost Benefit Analysis (CBA) should have been conducted to determine whether the project benefits adequately justifies the cost and the Return on Investment (ROI) should have been determined to assess whether it is a sound business proposal for the government to invest in. Moreover. it is evident that the proponent did not intend to generate revenues or recover the cost incurred in the project implementation. Furthermore.If the focus was more on the development of new network technologies and products that can be commercialized. in terms of Return on Investment.010. Although the main advantage of having a dedicated network should be that it has a high-speed connectivity (more faster than that of commercial ISPsnormal internet service) in order connect Philippines universities and other research institutions to the global research and education networks. the fact that the project’s original design did not include strategies for sustainability. it could have been the source of revenues or returns on the government’s almost P79M investment.719. this network connection should allow the creation of emerging network technologies (that need higher bandwidth for testing).new network technologies that can justify the cost. During the 5 year project. at presently DOST-ASTI is spending its yearly budget just to keep the network operational. An ROI study should have been done to see the returns of this project to its intended reciepientsrecipients. A Cost Benefit analysis between developing a dedicated network and commericialcommercial ISP shcould have been done.P R E G I N E T E V A L U A T I O N R E P O R T Figure 4 Project Budget – Approved. Spent Overall. these charges cannot adequately cover its operational expenses since at present. the budget covered the following areas: Personnel.

a number of partners became interested in the online learning management system. of Members Figure 5 Project Team Composition The project experienced high staff turn-over for the duration of its implementation. some activities were either discontinued or had to be redone/redesigned (due to lack of proper technical documentation). As a result. . Total No. One example of the activity that was discontinued was the deployment of the VClass System. Initially. there was nobody left to continue with its development and deployment.P R E G I N E T E V A L U A T I O N R E P O R T Staffing With its original 11-man project team (Figure 5). PREGINET experienced high staff turn-over rate during the implementation. One example of the activity that was discontinued was the deployment of the VClass System. However. when the whole development team resigned. As a result. The project was able to carry out a few trainings for institutions that expressed their interest to use it such as the Philippine Rice Research Institute (PhilRice) and the Benguet State University (BSU). and IPv6. some activities were either discontinued or had to be redone/redesigned (due to lack of proper technical documentation). The same case happened with the development and deployment of the Greenstone Digital Library.

. as DOST. there was nobody left to continue with its development and deployment. Efficient Sdocumentation processes and knowledge transfer mechanisms and protocols trict documentation should be properly in place. However. the attractiveness and value of these may vary from employee to employee. high staff turn-over rateTheir transfer to higher salary work became is inevitable. For example. In any case. a number of partners became interested in the online learning management system. a systematic and establishedized documentation process could have solvepossibly reduce the impact of d the problem disrupted and discontinued of abandoning or recreating programsprograms caused by and the likbecause of staff resignations. The same case happened with the development and deployment of the Greenstone Digital Library. during network problems. the Telof’s response time to address the problem was limited by factors such as: lack of capacity to properly address/fix the problem. it is a fact that competitive packages are available from the private sector. One factor that this project was not able to address sufficiently was the responsive and equitable reward & recognition system which can also cover salary. e. the . One factor that can be attributed to this was the unattractive compensation and benefits package offered to the researchers. Although there are also other motivating factors that ons keep researchers loyal to their employeran organization like: challenging. The project was able to carry out a few trainings for institutions that expressed their interest to use it such as the Philippine Rice Research Institute (PhilRice) and the Benguet State University (BSU).P R E G I N E T E V A L U A T I O N R E P O R T Initially. However. possible due to the Salary Standardization Law which could hinder ASTI from addressing this. In addition. a line agency is covered by the Salary Standardization Law (SSL)low salary coverage of ASTI for its researchers. and . In this case. if the technical problem involved an equipment that needed to be fixed/replaced/procured. status that maintain pPraise and recognition from peers in the scientific community. personnel. Initially. the regional access points were mainly the DOTC Telecommunications Office (Telof) facilities. equipment & time. Risk Management As the number of connected partners increased. adequate research facilities and . so did the need for a higher bandwidth as well as a more reliable network.and knowledge transfer policies within the project should have been part of the Standard Operation Process. monetary benefits or non monetary benefits like peer recognition. when the whole development team resigned. and IPv6. bonuses. interesting & important project assignments. which realizes the real value of these skilled human resources. and government standard operating procedures.

As a result. which in These new research areas iincludeclude among others. support next. major delays in operation and unsatisfactory service affected PREGINET’s partners and client. hosting and distribution . slow network connection. Education and Private firms in this country. The ultimate objective of the project should have been Advanced network services may beare necessary to createthe development of highly advanced. cutting. reveals a weakness in implementation or technology diffusion since it should have resulted in the use of advance network technologies by the Government. Product Development Successful R&D projects. Content and Applications development.ones improves products & services thus contributesservices thus contribute to higher productivity. It should provide customers (in the case of PREGINET. which are hardly cutting edge already mature technologies and readily available in the first in the market. better choices product performance & lower prices. Network Measurements. .edge. whether incremental or radical ones. d IpPv6. (especially those that need high speed internet) which is why PREGINET was created in the first place. Multimedia over IP. which should have been the foremost indicator of the project’s performance. However. tThe fact that the project did not result in a service or portfolio of products that are commercialized or new and even proven to be effective alternatives to existing technologies makes it apparent that the project failed in the area of product development. new products can beould have been commercialized and could possibly advance ICT development in theor make the Philippines a leader in Network Technology development.P R E G I N E T E V A L U A T I O N R E P O R T request had to go through the usual government procedures that caused major delays in the operation of the network.. These kinds of Risk should have been included in making contingency plans in the beginning of the project.These In turn. researchpresent endeavors of areas covered by PREGINET. academe and private industries) with varied choices.. lower costs & increased profits. its beneficiaries in government. As mentioned earlier. However.generation applications that is competitive in performance and price. first -in -the -market.

we will evaluate the post-evaluation stage of the project in view of the impact assessment conducted by DAP and the analysis will be focused on the following aspects: • • • • • Evaluation Team Evaluation Objectives Evaluation Methods Findings/ Results of DAP Evaluation DAP Recommendations Evaluation Team The DAP project team consisted of Elena Avedillo-Cruz. On the other hand. It outlines advantages and disadvantages of each option and examines possible synergies and trade-offs. Project Officer. It identifies and assesses the problem at stake and the objectives pursued. the Development Academy of the Philippines’ Center for Knowledge Management (DAP-CKM) was commissioned by ASTI to conduct the impact assessment of the PREGINET Project. It identifies the main options for achieving the objective and analyses their likely impacts in the economic. Thus." The Philippine Council for Advanced Science and Technology Research and Development (PCASTRD) under DOST was tasked to monitor the project accomplishments for the most part of project implementation based on the reports submitted by the project team in compliance with DOST policies and procedures. Managing Director. Cristina Bobis. Technical and Administrative .P R E G I N E T E V A L U A T I O N R E P O R T Post-Implementation Evaluation Stage C C “TA n impact assessment3 is a process aimed at structuring and supporting the development of policies. environmental and social fields. in this section. Sheryl dela Cruz.

In order to ensure the impartiality of the project evaluation. Determine conditions under which the projects are most effective. relevant and existing project documents were analyzed and the preliminary evaluation which resulted from the desk review was then validated using three sets of evaluation tools – the survey questionnaire. respectively could have been the contracting party for the evaluation. in the set-up. Just like the objectives set by the PREGINET project during the proposal stage. the contracting party to the agreement with the evaluating team should be independent from that of the project proponent or implementing unit. http://en. ASTI agreed to the following DAP evaluation objectives: 1. Evaluation Objectives In terms of impact. 4. Ascertain the contribution of the project to the overall program objectives. Supervising Fellow. Determine whether the stated project goals and objectives were achieved. Recommend possible projects that can be pursued in this field and suggestions to improve performance of such projects included. Evaluation Methods There were two major methods used in evaluating the projects’ impact – the desk review and the field assessment. 3. Identify the factors that contributed to the projects’ success or failure. Impact Assessment. As the funding and monitoring agencies for the project. 6. Caridad Aspiras. Technical Consultant. project proponent and implementer.org/wiki/Impact_assessment The result of the impact evaluation could have been considered objective because it was conducted by a third party organization. the above objectives are vague and should have also been patterned after the “SMART” objectives. However. the key informant interviews (KII) and the case studies.P R E G I N E T E V A L U A T I O N R E P O R T Staff. assessing the impact of the project in terms of R&D can be most appropriately expressed through quantifiable indicators such as the number of new network technologies developed. Moreover. Romero. 5. 2. DOST or PCASTRD. ASTI was the contracting party.wikipedia. and Segundo E. For the desk review. Assess the benefits of the projects. .

build up its partner institutions from the academe. delivery. In . advance the development of next generation technologies. it was able to establish a national REN. it was noted that only 29 (37%) of the 78 target respondents accomplished and returned the forms to DAP for data processing. This shows that the basis for the evaluation is not representative of the total population rendering the results to be dubious and questionable. and sustainability. b. and build the capability of human resources to support the achievement of these objectives. impact. testing and deployment of new network technologies. and research institutions with around 169 partnerships established as of August 2008 The project remained focused on the objectives it set to achieve. provide a platform that enabled the introduction. development.P R E G I N E T E V A L U A T I O N R E P O R T However. and c. develop new technologies. Thus. According to DAP: • The project objectives were attained because it was able to: a. which were used as basis for the evaluation (Figure 6). government. Figure 6 Retrieval Rate of Survey Questionnaire DAP Findings DAP evaluated the project in terms of four major areas – project design. applications and services and made these accessible for user communities. • The result that the “project objectives were attained” is not adequately justified given the fact that the accomplishments were not measurable.

and deployment of next generation network technologies. The intent for the PREGINET project was clear from the start. products and technologies promoted by ASTI: i. The overall assessment for PREGINET’s project design is summarized below: • a.P R E G I N E T E V A L U A T I O N R E P O R T addition. b. This was confirmed by the documents reviewed as well as the results of the field validation where the key informants consulted cited the following as bases for identifying the services. Needs and requirements of the user communities of PREGINET in line with the thematic areas being focused by ASTI. .” It will not be possible to fully assess and effectively evaluate a project’s impact without benchmarking it with “SMART” objectives. Focus of PREGINET is the development and adoption of advanced networking technologies that can benefit PREGINET partners. vi. PREGINET’s success as declared by DAP may be disputed. education. Services provided are identified based on the technologies and applications used to promote the development of advanced networks. and enterprise development v. only Pawikan was developed but was not successful as it was not fully commercialized. Anticipated and visible need of the country in network technologies. Need for the country to have an R&E network (REN) to be able to iii. except for the quantifiable and measureable targets that should have been set at the start. if assessed in the light of advancing the development of nextgeneration technologies. testing. namely environment. which is usually not pushed or probed by the private sector ii. Guided by the strategic objective “to establish a nationwide broadband network that will link research and education institutions for the development. DAP’s overall assessment that “the project was successful” is contradicted by their own finding that “there were no quantifiable and measureable targets set at the start of the project. government. Participate or have access with the research activities of other RENs around the world iv. Therefore.

disaster mitigation. the PREGINET project employed strategies that were found to be responsive to identified needs. Strengthen advocacy activities to promote the benefits of the PREGINET. The project. the lack of content to offer to partners should be addressed c. While there is an existing infrastructure. promotion of technological innovation and experimentation.” This is also contrary to fact that there was no concrete proof that the project led to the “acceleration of government connectivity. Seek the involvement of other stakeholders for advocacy such as the executive and legislative branches of the government. case studies. Actively solicit content from the partners. Another DAP finding is that “the PREGINET project employed strategies that were found to be responsive to identified needs. It seemed that the infrastructure was already provided by the project but content development was inadequate. and best practices. c) promotion of technological innovation and experimentation. In spite of the attainment of project objectives there is still a lingering perception that the partners have not fully realized or appreciated the benefits of the network. however. and bioinformatics”. In addition. b) development of IT manpower base. b. and d) creation of strategic partnerships among local institutions. The PREGINET project has been funded from the national government budget so it should be supported by the rest of the government sector to optimize its usefulness. PREGINET should speak for itself. and creation of strategic partnerships among local institutions. development of IT manpower base. telemedicine.P R E G I N E T E V A L U A T I O N R E P O R T services. It was during the latter part of the project that sustainability measures and mechanisms were incorporated into the design. agriculture. There is no substitute for better service. Partners who were highly satisfied with PREGINET's technologies. It should publicize its accomplishments in the form of stories. e. applications . and applications intended for distance education. d. DAP recommended the following: a. could have been optimized had it been promoted extensively. the project’s original design did not include strategies for sustainability. These areas are as follows: a) acceleration of government connectivity.” DAP Recommendations Based on the results of the evaluation.

among others and put in place efficient documentation processes and knowledge transfer mechanisms and protocols to reduce the project discontinuity caused by staff turn-over. this does not mean that financial tools and indicators are the only criteria for project selection because a well-balanced project portfolio requires inclusion of a few radical projects that would facilitate the long-term future viability of the organization. measurable. If competitive compensation and benefits package cannot be granted to skilled research staff.P R E G I N E T E V A L U A T I O N R E P O R T and services could give testimonials to inform and invite others to the network. b. interesting & important project assignments. f. Monitoring processes. attainable. Maintain the objectivity of the project evaluation by ensuring that the contracting party to the agreement with the evaluating team is independent from that of the project proponent or implementing unit. tools and mechanisms should be established and project evaluation should not be done only after project completion. Quantitative measures should be set to determine the project’s success. such as: . NPV should be used in project prioritization and selection. we recommend the following for future DOST projects: a. relevant and timebound (“SMART”). Education and .Number of patents generated . provide challenging. Tools for financial analysis like CBA and financial indicators such as ROI. Objectives should be specific. If DAP concentrated on assessing the project in terms of measurable objectives.Number of innovations commercialized .Number of refereed papers published . it is apparent that the project was not successful. he Advanced Science and Technology Institute (ASTI) of the Department of Science and Technology (DOST) commissioned the Development Academy of the Philippines’ Center for Knowledge Management (DAP-CKM) to conduct an impact evaluation of the VCTI-IT: Development of a High Performance Research and Education Network: The Philippine Research. Instead.Return on R&D investment d. adequate research facilities and equipment. e. c. the project milestones and targets are achieved. specifically the development of new technologies or products. for long-term projects. However. In view of this.Amount of external research grants obtained .Number of innovations developed . annual project audit and monitoring of accomplishments should be conducted to ensure that for each project phase.

2. 3. However. The preliminary evaluation which resulted from the desk review was then validated DAP Findings . the evaluation aimed to: 1. However. Technical and Administrative Staff. Managing Director. in the set-up. Caridad Aspiras. Project Officer. Evaluation Objectives In terms of assessing impact. The result of the impact evaluation could have been considered objective because it was conducted by a third party organization. The Philippine Council for Advanced Science and Technology Research and Development (PCASTRD) handled the monitoring of PREGINET and VCTI-RF projects since information technology and microelectronics were the priority areas of PCASTRD. Supervising Fellow. relevant and existing project and project-related documents were analyzed mainly to reconstruct the project design and to track the project results and accomplishments. DAP Evaluation Team and Time Frame The DAP project team consisted of Elena Avedillo-Cruz. 5. 6. 4.which is a component of the COMPETE program funded under DOST’s Grants-In-Aid (GIA). ASTI was the contracting party and at the same time the implementing agency of PREGINET. Romero. Determine conditions under which the projects are most effective. As the funding and monitoring agencies respectively. and Segundo E. For the desk review. Assess the benefits of the projects. Technical Consultant. Ascertain the contribution of the projects to the overall program objectives. Evaluation Methods There were two major methods used in evaluating the projects’ impact – the desk review and the field assessment.P R E G I N E T E V A L U A T I O N R E P O R T Government Information NETwork (PREGINET). PCASTRD only took actual charge of the project after the first implementation year. Determine whether the stated project goals and objectives were achieved. Identify the factors that contributed to the projects’ success or failure. Sheryl dela Cruz. Cristina Bobis. Recommend possible projects that can be pursued in this field and suggestions to improve performance of such projects included. DOST or PCASTRD should have been the contracting party to ensure the impartiality of the assessment.

Continuing R&D activities to keep abreast with current trends in advanced network technologies d. It remained focused on the objectives it set to achieve. Strengthened partnership with international research and education networks that enhanced connectivity to the research and education community c. which is usually not pushed or probed by the private sector viii. Advocacy on the benefits of the project However. development. delivery. government. e. According to DAP: The project objectives were attained because it was able to: advance the development of next generation technologies.g. after-sales service. and willing to spend extra hours to deliver the project e. and build the capability of human resources to support the achievement of these objectives. build up its partner institutions from the academe. certain conditions should be improved such as optimizing some internal systems. except for the quantifiable and measureable targets that should have been set at the start. and research institutions with around 169 partnerships established as of August 2008. International recognition and support that enabled the country to become integrated with the global research community b. The intent for the PREGINET project was clear from the start. open to learning. and policies for the use of PREGINET services. This was confirmed by the documents reviewed as well as the results of the field validation where the key informants consulted cited the following as bases for identifying the services. products and technologies promoted by ASTI: vii. e. • d. • • Some of the factors considered helpful to the project included: a. Participate or have access with the research activities of other RENs around the world . testing and deployment of new network technologies.P R E G I N E T E V A L U A T I O N R E P O R T DAP evaluated the project in terms of four major areas – project design. Need for the country to have an R&E network (REN) to be able to ix. provide a platform that enabled the introduction. Thus. Dynamic and young staff who were flexible. and sustainability. documentation of processes. and f. develop new technologies. • The overall assessment for PREGINET’s project design is summarized below. impact. applications and services and made these accessible for user communities. billing system. Anticipated and visible need of the country in network technologies. a. Support of DOST f. it was able to establish a national REN.

Actively solicit content from the partners. the project’s original design did not include strategies for sustainability. and enterprise development Focus of PREGINET is the development and adoption of advanced networking technologies that can benefit PREGINET partners. telemedicine. The project. services. Partners who were highly satisfied with PREGINET's technologies. Seek the involvement of other stakeholders for advocacy such as the executive and legislative branches of the government. Needs and requirements of the user communities of PREGINET in line with the thematic areas being focused by ASTI. b) development of IT manpower base. c) promotion of technological innovation and experimentation. testing. g. b. and applications intended for distance education. xi. namely environment. agriculture. In spite of the attainment of project objectives there is still a lingering perception that the partners have not fully realized or appreciated the benefits of the network. i. education. applications and services could give testimonials to inform and invite others to the network. the lack of content to offer to partners should be addressed h. These areas are as follows: a) acceleration of government connectivity. DAP Recommendations Strengthen advocacy activities to promote the benefits of the PREGINET. It should publicize its accomplishments in the form of stories. government. The PREGINET project has been funded from the national government budget so it should be supported by the rest of the government sector to optimize its usefulness. and bioinformatics”. There is no substitute for better service. xii. . PREGINET should speak for itself. Guided by the strategic objective “to establish a nationwide broadband network that will link research and education institutions for the development. In addition. It was during the latter part of the project that sustainability measures and mechanisms were incorporated into the design. While there is an existing infrastructure.P R E G I N E T E V A L U A T I O N R E P O R T x. f. and best practices. however. disaster mitigation. and d) creation of strategic partnerships among local institutions. and deployment of next generation network technologies. could have been optimized had it been promoted extensively. the PREGINET project employed strategies that were found to be responsive to identified needs. j. case studies. It seemed that the infrastructure was already provided by the project but content development was inadequate. Services provided are identified based on the technologies and applications used to promote the development of advanced networks.

e. certain conditions should be improved such as optimizing some internal systems. What are they doing about this? Is there a target no. Through the development and testing of applications in new areas. of institutions per year that will become a partner? No measurable objective What new technologies were developed? However. The project has definitely opened opportunities for ICT development and advancement in the country. There is an expressed need. after-sales service. The growing appreciation of the TELCOs on the importance of PREGINET role is itself recognition of the benefits the project has brought about in helping the advancement of network technology . PREGINET has made its mark with the TELCOs. during the data Final Report 26 gathering phase to further strengthen the R&D group of PREGINET particularly on identifying services and technologies that would be beneficial. documentation of processes. there is really a need to promote the benefits of the network. several network and information and communication technologies have been enhanced with PREGINET. the technical competency of the staff involved in the implementation of PREGINET is being continuously upgraded through exposure and participation in gatherings where application of the ICT technologies are tested and demonstrated. however.g. and policies for the use of PREGINET services. Thus. What are they doing about this? Further. billing system.P R E G I N E T E V A L U A T I O N R E P O R T ANALYSIS How is the performance of the network? Did DAP conduct an interview with the users of the REN? How was the service offered to its partners? Was it marketed aggressively? Are there LGUs using the network? What is the criteria for becoming a partner? What were the actual applications developed? But public appreciation for the PREGINET is still somewhat low.

In terms of cost-efficiency. This is based on the fact that if we were to evaluate the project and focus on the R&D aspect. next generation network technologies for the Philippine Government.  . competitive rate or better performance. we arrived at the conclusion that the project was unsuccessful and did not have a significant impact on R&D. What does this mean? 4 Conclusion -B CONCLUSIONased on our analysis of the PREGINET Project. but rather adpatation of foreign technologies. as it does not provide unique features. The project did not generate enough revenues to maintain its operations or at least recover the government’s investment. It cannot be considered a success in advancing new. there were no successful new technologies developed and the research activities were mere adoption of available and matured technologies that does not justify the P79M project cost. if there were any. The project lacks effective monitoring tools and mechanisms to determine the R&D output of partner-beneficiaries such as research papers published and patents granted. •  • • • • Research done over the project duration are not classified as advanced network technologies. a monitoring tool called Pawikan was developed but it was not commercialized rendering the project failed in terms of R&D output. PREGINET services have no competitive advantage over other commercially-available services. the following will be the results: Only one network technology. the project failed as evident in the fact that during the 7-year project duration.P R E G I N E T E V A L U A T I O N R E P O R T and strengthening the network infrastructure.

lack of commercialization and there is still preference for private network providers. research. The lack of measures and Key performance indicators for R&D.  In relation to this. 5 List of References Appendix Appendix 1. telemedicine. But seeing how there was a lack of technology diffusion.P R E G I N E T E V A L U A T I O N R E P O R T Academe and Private firms. it cannot justify DOST’s return of Investment on the project in this 7 year project. led to increase turn-over of staff. Multimedia over IP . patents. there should have been new products and services that resulted from R&D that can benefit society or prove to be a better alternative for Filipino users.the standard for the next generation Internet. List of Research Areas duingduring PREGINET project • • IPv6 .which will deploy H.  Low monetary benefits affect the work motivations of Researchers and thus. Morever. and other uses of videoconferencing and multimedia streaming. An offshoot of high turnover and lack of SOP regarding documentation and knowledge transfer is the non-deployment of some projects when researchers resign. led to non measurement of R&D output. both for PREGINET and its PARTNERS. papers. PREGINET was not successful in this aspect.323 and multimedia (voice and video) streaming as standard network services for the research and academic community to provide distance learning. . IPv4. and which addresses the limitations of the current Internet protocol.

which deals with the establishment of the PREGINET nationwide network and the study of different wire and wireless broadband network technologies.which involve the use of network monitoring tools and systems to measure the performance of various services and applications under test as well as to monitor network health. hosting and distribution – which includes the provision of high-availability servers and computing systems for scientific applications that shall be made available for use by the partner institutions for their projects over PREGINET. Appendix 2.P R E G I N E T E V A L U A T I O N R E P O R T • • • Network Measurements . and Content and applications development. PREGINET Partners . Network Infrastructure .

Sign up to vote on this title
UsefulNot useful