H2020 – SOCIETAL CHALLENGES – Europe in a changing world - Inclusive, innovative and reflective societies

DT-TRANSFORMATIONS-02-2018 – Transformative impact of disruptive technologies in public services – Research and

Innovation Action (RIA)

Decentralised Qualifications' Verification and Management for Learner

Empowerment, Education Reengineering and Public Sector Transformation

D7.6 QualiChain Pilot Evaluation and Lessons Learnt

WP7 – QualiChain Services Piloting, Evaluation and Impact

Workpackage:
Assessment

Nadia Politou (ATOS), Ingo R. Keck (TIB), Christos Kontzinos (NTUA),

Authors: Alexander Microyannidis (OU), Andreia Pimenta (SRS), Sergio Guerreiro
(INESC-ID), ASEP team (HeP/ASEP), Sofia Paredes (AMA)

Status: Final

Due Date: 31/12/2021

Version: 1.00

Submission Date: 14/01/2022

Dissemination Level: PU *

Disclaimer:
This document is issued within the frame and for the purpose of the QualiChain project. This project has received funding from the European Union’s
Horizon2020 Framework Programme under Grant Agreement No. 822404. The opinions expressed and arguments employed herein do not necessarily
reflect the official views of the European Commission.
This document and its content are the property of the QualiChain Consortium. All rights relevant to this document are determined by the applicable laws.
Access to this document does not grant any right or license on the document or its contents. This document or its contents are not to be used or treated
in any manner inconsistent with the rights or interests of the QualiChain Consortium or the Partners detriment and are not to be disclosed externally
without prior written consent from the QualiChain Partners. Each QualiChain Partner may use this document in conformity with the QualiChain Consortium
Grant Agreement provisions.
(*) Dissemination level.-PU: Public, fully open, e.g. web; CO: Confidential, restricted under conditions set out in Model Grant Agreement; CI: Classified,
Int = Internal Working Document, information as referred to in Commission Decision 2001/844/EC.
QualiChain D7.6
QualiChain Pilot Evaluation and Lessons Learnt

QualiChain Project Profile

Grant Agreement No.: 822404

Acronym: QualiChain

Decentralised Qualifications' Verification and Management for

Title: Learner Empowerment, Education Reengineering and Public
Sector Transformation

URL: https://qualichain-project.eu

Start Date: 01/01/2019

Duration: 36 months

Partners

NATIONAL TECHNICAL UNIVERSITY OF ATHENS, Decision

Greece
Support Systems Laboratory, (NTUA) Co-ordinator

ATOS SPAIN SA (ATOS) Spain

FRAUNHOFER GESELLSCHAFT ZUR FOERDERUNG DER

Germany
ANGEWANDTEN FORSCHUNG E.V. (IAIS)

KNOWLEDGEBIZ CONSULTING-SOCIEDADE DE
Portugal
CONSULTORIA EM GESTAO LDA (KBZ)

United
THE OPEN UNIVERSITY (OU)
Kingdom

TECHNISCHE INFORMATIONSBIBLIOTHEK (TIB) Germany

INSTITUTO DE ENGENHARIADE SISTEMAS E

COMPUTADORES, INVESTIGACAO E DESENVOLVIMENTO Portugal
EM LISBOA (INESC-ID)

AGENCIA PARA A MODERNIZACAO ADMINISTRATIVA IP

Portugal
(AMA)

HELLENIC PARLIAMENT (HeP)

Greece
SUPREME COUNCIL FOR CIVIL PERSONNEL SELECTION
(ASEP) – Linked Third Party

Secretaria Regional de Saúde e Proteção Civil (SRS) Portugal

UNINOVA-INSTITUTO DE DESENVOLVIMENTO DE NOVAS

Portugal
TECNOLOGIASASSOCIACAO (UNINOVA)

2
QualiChain D7.6
QualiChain Pilot Evaluation and Lessons Learnt

Document History
Version Date Author (Partner) Remarks/Changes

0.10 24/8/2021 Nadia Politou (ATOS) Table of Contents

Christos Kontzinos, Kostas

0.11 3/11/2021 Comments on the TOC
Ergazakis (NTUA)

0.12 3/11/2021 Nadia Politou (ATOS) Updated TOC

Section 3.3 Staffing the Public Sector –

0.20 12/12/2021 ASEP team (HeP/ASEP)
The Case of Greece.

Section 3.1 3.1 Supporting Lifelong

0.21 15/12/2021 Alexander Microyannidis (OU)
Learning.

Christos Kontzinos, George Section 3.2 University Process Optimisation

0.22 15/12/2021 Trachanas, Stavros Skalidakis through Smart Curriculum Design and
(NTUA) Student Accreditation.

Section 3.5 Provision of Recruitment and

0.23 22/12/2021 Andreia Pimenta (SRS)
Competency Management Services.

Section 3.4 Staffing the Public Sector – The

0.24 31/12/2021 Sérgio Guerreiro (INESC-ID)
Case of Portugal.

Document harmonisation. Distribution to

0.30 7/01/2022 Nadia Politou (ATOS)
TIB for the preparation of section 4.

Section 4 Methodological Adoption

0.40 11/01/2022 Ingo R. Keck (TIB) Guidelines for the Utilisation of the
QualiChain Platform.

Refinements. Introduction, Conclusions and

0.50 12/01/2022 Nadia Politou (ATOS) Executive Summary sections. Distribution
for peer review.

Contributions to Section 3.4 Staffing the

0.51 13/01/2022 Sofia Paredes (AMA)
Public Sector – The Case of Portugal.

Christos Kontzinos, Ourania

0.60 13/01/2022 Peer review feedback
Markaki (NTUA)

0.61 13/01/2022 Ingo R. Keck (TIB) Peer review feedback

0.62 13/01/2022 Pedro Oliveira (KBZ) Peer review feedback

0.70 14/01/2022 Nadia Politou (ATOS) Incorporation of feedback

Panagiotis Kokkinakos, John

1.00 14/01/2022 Final version
Psarras (NTUA)

3
QualiChain D7.6
QualiChain Pilot Evaluation and Lessons Learnt

Executive Summary
The current document provides the outcomes of task 7.7 “QualiChain Pilots Evaluation, Lessons Learnt
and Impact Assessment”, which is a set of lessons learnt from the execution of the QualiChain pilots
and guidelines for the future adoption of the QualiChain solution. The lessons learnt were extracted
from the feedback of the users who experimented with the QualiChain platform during the lifetime of
the project, in the context of the five different pilots of the project in three different countries; Greece,
Portugal and the UK.
The lessons learnt were derived from the interactions with the users and the evaluation results as
reported in D7.5 QualiChain Pilots Execution Documentation, Final Release [1]. The lessons learnt from
the pilots helped to identify a set of methodological adoption guidelines for the adoption of QualiChain
after the end of the project. These have been classified into four main categories:
• Strong points of QualiChain: such as ePortfolio that contains the digital identity of learners and
verified qualifications, data sovereignty and personal data protection, support for informed
decision making, among others.
• Weak points of QualiChain: such as weak knowledge about the social aspects of increased
mutual certification of soft skills or difficulties of the users with the user interface or difficulties
of the users using the user interface or the MCDSS.
• Integration with external systems: within the context of the project, QualiChain has been
integrated only with FenixEdu1 for the purposes of the “Staffing the Public Sector – The case
of Portugal” pilot. Nevertheless, in other pilots, integration with external systems would be of
great value. For example, in the case of “Staffing the Public Sector – The case of Greece” pilot,
it would be of great benefit to integrated with ASEP´s system or in the case of “University
Process Optimisation through Smart Curriculum Design and Student Accreditation”, students
requested, if possible, to integrate it with further University systems so that it can be a holistic
tool for the University.
• External environment: such as the high regulation that exists in the public sector and academia
which makes it hard to introduce innovative solutions and often requires the involvement of
too many stakeholders.
The report concludes with a set of guidelines specific to the above-mentioned categories so that they
can serve for the future adaptations of the solution and lay the path for a successful public adoption of
QualiChain.

1
https://fenixedu.org
4
QualiChain D7.6
QualiChain Pilot Evaluation and Lessons Learnt

Table of Contents
1 Introduction ............................................................................ 8
1.1 Purpose and Scope........................................................................................ 8
1.2 Structure of the Document ........................................................................... 8
1.3 Relationship with other Documents ............................................................. 8
1.4 List of Abbreviations ..................................................................................... 9

2 Lessons learnt from the pilot evaluation activities ............... 10

2.1 Supporting Lifelong Learning ..................................................................... 10
2.1.1 Lifelong learners require support in several aspects of their learning journey
10
2.1.2 The education community requires guidance and training on the use of
decentralisation technologies ......................................................................... 11
2.2 University Process Optimisation through Smart Curriculum Design and
Student Accreditation ................................................................................. 11
2.2.1 CV gathering from students and data protection ....................................... 11
2.2.2 Stakeholder engagement for the pilot ....................................................... 12
2.2.3 QualiChain integration in the ECE school .................................................. 13
2.2.4 Curriculum updates and optimisation ........................................................ 13
2.2.5 Personalised services for students ............................................................ 14
2.2.6 Smart badge accreditation ....................................................................... 14
2.2.7 Multi-criteria decision support systems ..................................................... 15
2.2.8 QualiChain platform as a whole ................................................................ 15
2.3 Staffing the Public Sector – The Case of Greece ........................................ 16
2.3.1 Implementation issues ............................................................................ 16
2.3.2 Using QualiChain in a production environment .......................................... 16
2.3.3 Data protection ....................................................................................... 16
2.3.4 Multi-Criteria Decision Support System (MCDSS) ....................................... 17
2.3.5 QualiChain integration with ASEP system and procedures .......................... 17
2.3.6 QualiChain Platform: Corner stone with great potential.............................. 18
2.4 Staffing the Public Sector – The Case of Portugal ...................................... 18
2.4.1 Learning curve for the technologies related with blockchain ....................... 18
2.4.2 End users lack knowledge related to blockchain technology ....................... 18
2.4.3 Clear guidelines definitions provided to the end users ............................... 18
2.4.4 Few standards and regulations available ................................................... 18
2.4.5 Lack of integration with eIDAS authentication services .............................. 19
2.4.6 The UI could be more intuitive and user-friendly ....................................... 19
2.4.7 Diploma verification could be more automatic ........................................... 19
5
QualiChain D7.6
QualiChain Pilot Evaluation and Lessons Learnt

2.5 Provision of Recruitment and Competency Management Services ............ 19

2.5.1 Introduction of a human resources platform in public health administrative
procedures ................................................................................................... 19

3 Methodological Adoption Guidelines for the Utilisation of the

QualiChain Platform .............................................................. 21
3.1 Strong points of QualiChain ........................................................................ 21
3.2 Weak points of QualiChain .......................................................................... 22
3.3 Integration with external systems ............................................................. 22
3.4 External environment ................................................................................. 23

4 Conclusions ........................................................................... 24
Annex I: References .................................................................... 25

6
QualiChain D7.6
QualiChain Pilot Evaluation and Lessons Learnt

List of Figures

Figure 2-1: Results from the consultation with stakeholders in the context of the OU pilot. .............. 10

List of Tables

Table 1-1: Table of Abbreviations .................................................................................................. 9

7
QualiChain D7.6
QualiChain Pilot Evaluation and Lessons Learnt

1 Introduction

1.1 Purpose and Scope

The purpose of the current report is to document the lessons learnt from the piloting activities
performed by the five QualiChain pilots performed within the context of WP7 “QualiChain Services
Piloting, Evaluation and Impact Assessment”. The report builds on the evaluation performed during the
last phase of the QualiChain pilots and reported in D7.5 QualiChain Pilots Execution Documentation,
Final Release [1]. The report also provides a set of methodological adoption guidelines for the future
utilisation of the QualiChain platform.

1.2 Structure of the Document

The deliverable is structured as follows:
Section 1 provides the introduction, explaining the purpose and scope of the document, its structure
and its relation to other WPs and tasks of the project. It also includes the list of abbreviations used
throughout the document.
Section 2 documents all the lessons learnt per pilot based on the pilot evaluation activities.
Section 3 provides the methodological adoption guidelines for the future utilisation of the QualiChain
platform.
Section 4 provides the conclusions of the deliverable.
Annex I lists the references.

1.3 Relationship with other Documents

This deliverable is the last of a series of reports delivered within WP7 throughout the project. It
accompanies D7.5 QualiChain Pilots Execution Documentation, Final Release [1] which includes the
actual evaluation results from the last phase of the QualiChain pilots. The deliverable also closely relates
to the following deliverables:
• D2.2 QualiChain Stakeholders’ Requirements and Use Cases [2], which describes the
stakeholders, their requirements, and the covered use cases.
• D7.1 Pilots Preparation Handbook [3], which is a guide for the setup and execution of the pilots
in QualiChain.
• D7.2 Pilots Readiness Documentation [4], which describes the test cases and evaluation criteria
of the pilots in detail.
• D7.3 QualiChain Pilots Execution Documentation, Release I [5], which updates the test cases
and evaluation criteria of the pilots.
• D7.4 QualiChain Pilots Execution Documentation, Release II [6], which is the second update of
the test cases and evaluation criteria of the pilots.

8
QualiChain D7.6
QualiChain Pilot Evaluation and Lessons Learnt

1.4 List of Abbreviations

The following Table 1-1 includes all abbreviations used in the document.

Table 1-1: Table of Abbreviations

Abbreviation Text

API Application Programming Interface

BOC Badged Open Course

CV Curriculum Vitae

ECE Electrical and Computer Engineering school

GDPR General Data Protection Regulation

HEI Higher Education Institution

MAUT Multi-attribute Utility Method

MCDSS Multi-Criteria Decision Support System

UI User Interface

UK United Kingdom

9
QualiChain D7.6
QualiChain Pilot Evaluation and Lessons Learnt

2 Lessons learnt from the pilot evaluation activities

This section provides the lessons learnt as a result of the pilot evaluation activities. These have been
extracted from the experience of the execution of the QualiChain pilots, the evaluation results and the
input from the users who experimented with the QualiChain platform in the different pilot settings.
There is one sub-section per pilot. Each lesson learnt includes a title, a description and one or more
categories.

2.1 Supporting Lifelong Learning

This sub-section is dedicated to the lessons learnt from the operation and execution of the pilot entitled
“Supporting Lifelong Learning” that was implemented by the Open University (OU) in the UK.

2.1.1 Lifelong learners require support in several aspects of their learning journey
During the first year, we collected requirements for the OU pilot via a series of consultation workshops,
the outcomes of which have been reported in D7.3 [5]. During this consultation, a number of lifelong
learning aspects emerged, in which the stakeholders of the OU pilot require support. These aspects are
summarised in Figure 2-1. First of all, stakeholders pointed out the need for ePortfolios to aggregate
immutable formal and informal qualifications that will be easily validated by employers and educational
institutions. Stakeholders highlighted the need to be guided on how to build lifelong learning pathways
in order to achieve their learning goals. Acquiring micro-credentials can help lifelong learners achieve
these goals by studying short online courses and earning professional or academic credentials.
Stakeholders also indicated that they need a comprehensive overview of the job market and the latest
market trends, so that they can make informed decisions about the next steps in their careers. Finally,
data ownership and privacy requirements were deemed quite important by stakeholders. It was
highlighted that lifelong learners should own their digital identity and the data in their
ePortfolio. Additionally, they should be able to control who accesses their identity and their ePortfolio,
which data are accessed and for how long. Most of these aspects have been addressed by the OU pilot
via decentralised management and verification of qualifications, personalised job and course
recommendations offered to lifelong learners, as well as decentralised mechanisms for data ownership
and privacy.
Category: Multiple categories, e.g., technological, organisational, policy, and more.

Figure 2-1: Results from the consultation with stakeholders in the context of the OU
pilot.

10
QualiChain D7.6
QualiChain Pilot Evaluation and Lessons Learnt

2.1.2 The education community requires guidance and training on the use of
decentralisation technologies
There is currently a hype around Blockchain, mostly due to the emergence of Bitcoin and other
cryptocurrencies. However, very few members of the education community are aware of the potential
of decentralisation for transforming education. During our consultation and piloting activities with a
wide range of stakeholders from the education community, it became quite clear that there is a need
to provide guidance and training to the education community regarding the use of
Blockchain in education and specifically in lifelong learning. Towards addressing this need, the
OU pilot has produced the Badged Open Course (BOC): “Decentralising Education Using Blockchain
Technology”. This self-paced course introduces Blockchain technology and its potential for
decentralising and transforming education. The course also presents the work carried out by the
QualiChain project for the decentralised management and verification of education and employment
qualifications, specifically in the context of lifelong learning. Finally, the course offers an opportunity to
learners to try the online platform developed by QualiChain and earn a Smart Badge. The course is
available on the OU’s OpenLearn Create platform and is licensed under CC BY-NC-SA 4.0, which allows
the reuse and adaptation of the course materials for non-commercial purposes. Upon completion of the
course, learners earn a free statement of participation. As reported in D7.5, the participants of the OU
pilot have provided positive feedback about the course and the learning resources that it offers. The
course will remain available as a sustainable result beyond the lifetime of QualiChain, in order to
continue educating the public about decentralisation.
Category: Primarily educational and technological.

2.2 University Process Optimisation through Smart Curriculum

Design and Student Accreditation
This sub-section is dedicated to the lessons learnt from the operation and execution of the pilot entitled
“University Process Optimisation through Smart Curriculum Design and Student Accreditation” that was
implemented in the Electrical and Computer Engineering school (ECE) of the National Technical
University of Athens (NTUA). The QualiChain platform was presented and used by 90 students
(undergraduate and Ph.D.) of the school and 11 professors. The feedback that was received by the
pilot users and the conclusions that were drawn from this pilot led to the lessons reported in the
following sections.

2.2.1 CV gathering from students and data protection

At the beginning of this pilot, the pilot leader requested through various channels (classroom, social
media, university websites, e-mails) CVs from undergraduate and Ph.D. students that would be
analysed to draw conclusions on the average skill level of students. Since the CVs that were requested
represented personal data, according to the legal strategy of the project that is compliant to the GDPR,
they had to be accompanied by a consent form, as this was defined in the deliverables of WP1. Based
on the legal obligations of QualiChain, this consent form required a signature from an official
administrative body (such as a police precinct) which was a tedious process for students. For that
reason, at the beginning of the pilot’s operation, participation of students was low with regards to CV
gathering. During the 2nd year of QualiChain’s operation and during the first wave of the Covid-19

11
QualiChain D7.6
QualiChain Pilot Evaluation and Lessons Learnt

pandemic, additional digital services from the Greek government allowed this process to be performed
online. This shift in administrative processes greatly increased the number of CVs sent from students.
Before this change, the pilot had only managed to gather around 10 CVs, while after the creation of a
digital solution, 60 additional CVs were sent from students in a very short time period. It is evident that
in our modern society, in which ICT technologies provide novel solutions for various everyday problems
and tasks, the effort to make a shift from physical to digital administrative processes needs
to be intensified. Nowadays, citizens are always “connected” either via a personal computer or
through a smartphone and the administrative processes of Greece need to reflect and leverage that
fact to create new and innovative tools that help citizens perform everyday tasks. This need was also
magnified by the Covid-19 pandemic and the quarantine situation. In addition, this activity validated
QualiChain’s legal strategy and provided some perspective regarding the GDPR. It was proven that the
GDPR should not be viewed as a barrier when it comes to developing technical solutions but rather as
a set of guidelines to protect personal data. More information concerning QualiChain’s GDPR compliant
strategy can be found in the deliverables of WP1.
Category: Procedural, Technological, Legal

2.2.2 Stakeholder engagement for the pilot

While this pilot use case presented the QualiChain platform to over 100 users, which were enough to
receive feedback, draw useful conclusions and address the KPIs, the restrictions imposed by the
pandemic and the new reality of digital classrooms created problems when it came to engaging
stakeholders of the school. While the digital classroom paradigm is necessary when it comes to
safeguarding public health, it was concluded that digital engagement and participation suffer compared
to physical. While the pilot leaders made great effort to engage platform users by leveraging every
channel available to them, the response rate was not the one that was anticipated. In addition, digital
workshops and focus groups suffer greatly compared to physical ones as they impose
restrictions when it comes to the number of participants, as well as the level of
engagement and communication among them. In the initial stages of the pilot, it was made
possible to validate the pilot concept in impromptu workshops in the classroom in front of large groups
of students. This is not possible when most courses are taught online. Another conclusion drawn,
when it comes to online teaching, was the increase of the workload stemming from almost
every course. Students found themselves with more obligations and assignments, which made it
harder to engage in extracurricular activities. This was also manifested in the difficulties we faced to
organise the workshops by finding a common time that would suit the participants. While these
challenges were largely met in this pilot, this was done by spending great effort for each participant. It
is a fact that when it comes to research projects, the pilot execution phase always requires the
participation of stakeholders that are defined by each respective project. If the Covid-19 situation
persists along with the paradigms of remote working/learning, current and future projects need to come
up with new strategies when it comes to engaging users and piloting their solutions (engaging users
from earlier stages, increasing number of activities etc.). In any case, it is believed that these changes
should be reflected in the effort declared and spent for piloting.
Category: Methodological

12
QualiChain D7.6
QualiChain Pilot Evaluation and Lessons Learnt

2.2.3 QualiChain integration in the ECE school

While the QualiChain platform was successfully piloted in the ECE school, it is far from being considered
officially integrated in the school’s processes. As most Higher Education Institutions (HEIs), NTUA also
operates by following strict administrative processes that are set out by its various administrative
bodies. This creates many burdens when it comes to considering and implementing updates and
changes in the school’s operation, even in the cases in which the changes will have a positive impact.
For example, the introduction of new courses in the school’s curriculum needs to be agreed upon by
the council of professors. While this council meets several times a year, the thematic is usually
overflowing with administrative issues and the addition of new courses is often scrutinised by the
professors. In addition, any changes in the school’s processes must be agreed upon both by the council
of professors as well as the school’s secretariat. QualiChain would bring about innovative, but also
radical changes, when it comes to the everyday operation of the school, such as introducing a new
evaluation system for students and professors via smart badge accreditation. Even if a change or update
is approved by professors and the secretariat, there is also an administrative student body that is highly
political and has the power to veto some decisions. The current situation in the university (which also
reflects the reality in most Greek universities) is counterproductive when it comes to implementing
changes and introducing innovative practices in a school’s operation. It is strongly believed, as
concluded by the pilot’s operation, that university administration in Greece should become
simpler, more agile, and less resistant to change.
Category: Organisational, Procedural, Policy

2.2.4 Curriculum updates and optimisation

One of the key objectives of this pilot was to analyse the curriculum of the ECE school, compare it to
the skills that are in high demand by the labour market and suggest changes that would bridge the gap
between skills taught at the school and the requirements of professions that school graduates are likely
to pursue. This process was performed for the domain of software engineering that represents many
of the school’s courses. The tools developed by the technical team succeeded in identifying missing
skills and even provided recommendations concerning courses in which those skills could be introduced.
Some of the professors that participated in the pilot teach courses for which updates were suggested.
The professors found the updates to be suitable and useful for the development of their students’ skills
and were positive in introducing them in their courses. However, introducing new skills to a course and
in the curriculum is a more complicated process. Professors would have to consider the rest of the skills
that are being taught, the overall workload and existing assignments, as well as how much the course
would change by introducing an additional skill in its syllabus. This pilot made it evident that the process
of updating a university curriculum is not simple and requires the effort of all stakeholders, especially
the professors and the school’s administrative bodies. In fact, simply adding new and innovative skills
in a course would result in increased workload, additional assignments and more time required by both
students and professors. If a university wants to update its curriculum, the process followed must be
robust and organised. Through the conclusions drawn from this pilot, it would be suggested that such
a process begins by identifying the skills that are being taught but are also considered
obsolete when it comes to the labour market requirements. A distinction should be made
between obsolete skills (e.g., AVR microprocessor and Pascal programming languages) and introductory
skills (e.g., 8085 microprocessor and C programming language), recognising that students need to learn
the basics before moving on to more advanced skills. Such a process would then result to several

13
QualiChain D7.6
QualiChain Pilot Evaluation and Lessons Learnt

“holes” in the current courses that would need to be filled by more relevant skills. In addition, new
courses could be introduced to cover some of those skills as was suggested by some professors who
participated in the pilot. In conclusion, it can be said that the QualiChain platform provides the tools
that a university would require for curriculum updates (analytics and decision support), but the actual
process of curriculum update needs to be holistically undertaken by the administrative bodies of the
school.
Category: Organisational, Horizontal, Policy

2.2.5 Personalised services for students

The QualiChain platform offered, among other, several functionalities that are personalised to students
of the ECE school, such as course and thesis selection, and personalised recommendations regarding
skills and courses. While most students had a positive opinion regarding these personalised services,
we also received feedback from them regarding their expectations from such a platform. Several
students were disappointed that the platform was not connected to the school’s administrative
platforms and as a result they had to select the courses that they are attending all over again. Moreover,
they would like to see other functionalities in QualiChain regarding their everyday life such as a calendar
for courses and assignments, connection of the platform with the school’s forum (that is being managed
by students) and more notifications concerning school announcements. In addition, some students
thought that QualiChain should be linked with other platforms, such as LinkedIn, so that their CV and
skills could be transferred from one platform to the other. While overall, students were quite satisfied
with QualiChain and the combination of innovative technologies that it offers, the lack of connection
with similar platforms, combined with the fact that QualiChain’s UI is not as intuitive as
they would like in some functionalities could result in some users losing interest in the
platform. This is a conclusion that should be considered by QualiChain as well as other similar research
projects and initiatives. As mentioned above, the integration of QualiChain with the school’s existing
systems presents some realistic difficulties. As such, radical administrative changes are required to pave
the road for solutions like QualiChain to be implemented in their respective environments.
Category: Technological, Horizontal

2.2.6 Smart badge accreditation

The smart badge accreditation functionality of the QualiChain platform was the most well received by
the pilot’s stakeholders. Smart badges offer verifiable and immutable validation for skills gained and
tasks completed and as such, they were positively evaluated by both students and professors. The
integration of the smart badge functionality in the everyday life of the university could potentially lead
to some resistance and complaints by some professors and groups of students that do not agree with
this type of student evaluation. Some students thought that such a process could lead to the creation
of unfair leaders in the cases of exceptional or popular students. Of course, such an issue would be
easy to solve by setting an upper limit to how many instances of a smart badge can be awarded to a
student by other students. Professors on the other hand, can be trusted to be impartial. In addition,
based on the feedback that we received, it seems that despite the explanations provided, students and
professors still had some misunderstandings concerning the role of blockchain in the platform. Some
thought that the right to data deletion could not be respected despite the fact that no personal data
are stored in the platform’s blockchain. Other failed to see the merit of having blockchain-verified smart
badges in their profiles, given that their understanding of the technology is lacking. It is strongly

14
QualiChain D7.6
QualiChain Pilot Evaluation and Lessons Learnt

believed that knowledge around blockchain should be increased via inclusion of blockchain
in the curriculum, special lectures, and platform exhibitions, as well as cooperation of HEIs
with national and European blockchain initiatives. On another note, the smart badge
functionality provided the opportunity to verify soft skills as well. Skills like teamwork, ability to
communicate, leadership etc. can manifest during presentations and collaborative assignments. While
these skills are also requested by the labour market, job seekers are having difficulties in proving them
to prospective employers.
Category: Technological, Horizontal, Policy

2.2.7 Multi-criteria decision support systems

The Multi-Criteria Decision Support System (MCDSS) provided platform users with a tool that can help
them make more informed decisions, by ranking alternative options based on criteria that they set. In
the workshops that took place under the context of this pilot, the explanation of this functionality took
some time, since most users were not familiar with it. It should also be noted that the explanation only
covered one of the four MCDSS methods included in the tool. While, in the end, the users understood
how to use the tool and verified its usefulness they also provided feedback on how to make it easier to
use, especially for novice users. Students thought that if this functionality also included a few examples
of how to use it, they would find it easier to use without requiring explanation. In addition, the tool
could include several criteria that users could select automatically, depending on the type of analysis
that they want to perform. When it comes to criteria like “difficulty” and “workload”, while they are
somewhat subjective, the tool could have a memory of previous analyses so that the value of said
criteria could be the average value that users set in previous analyses. The values of some criteria could
also be calculated based on the answers to the questionnaires that students are called to fill in after
the completion of a course. However, it seems unlikely that a lot of professors would provide that
knowledge or even analyse it and add it to the platform, unless QualiChain was officially integrated in
the university’s processes. Such changes would significantly reduce the time it takes users to calculate
the values of each criterion (e.g., state of the art concerning a course). In fact, leveraging past
analyses to calculate the average values would mean that the more the tool is being used
these average values would be even more realistic. The aforementioned changes are considered
critical for the success of the MCDSS tool, and they could be combined with a tutorial video that
sufficiently explains the functionalities of the tool and its various MCDSS methods.
Category: Methodological, Technological

2.2.8 QualiChain platform as a whole

Overall, it can be said that the implementation of the QualiChain platform under the context of this pilot
was successful, as can also be seen by the positive feedback received via the questionnaires. QualiChain
presented to stakeholders of higher education, a combination of technologies that they have not seen
before under the context of education. It can be said that leveraging innovative technologies can lead
to new and original platforms and tools that meet real needs and provide out of the box solutions. From
a technical standpoint, combining those innovative technologies does not always lead to the most robust
solutions. However, it is imperative that such approaches are explored in order to develop the services
of the future. On another note, research projects like QualiChain are critical in bringing people in contact
with some of those technologies which can lead in increased trust. For example, while blockchain is still
considered by most as synonymous of cryptocurrencies, QualiChain proved that it is largely

15
QualiChain D7.6
QualiChain Pilot Evaluation and Lessons Learnt

underutilised in other domains (health, education etc.), as a solution for dealing with private and
sensitive data. Especially, in the Covid-19 and post-Covid-19 era, with a new reality being shaped when
it comes to learning and working, functionalities like degree verification, smart badge
accreditation, and decision support could and should be at the forefront of future solutions.
Category: Horizontal, technological

2.3 Staffing the Public Sector – The Case of Greece

This sub-section is dedicated to the lessons learnt from the operation and execution of the pilot entitled
“Staffing the Public Sector – The Case of Greece”, that was implemented by the Hellenic Parliament
and its linked third party ASEP.

2.3.1 Implementation issues

The implementation of the platform took a lot of effort from the developers, many meetings, and a lot
of discussion between partners. It was also necessary to make many compromises between the
partners, so that all the pilots could be implemented, as they had been specified in the earlier stages
of the project. It is important to resolve integration issues as early as possible in a project
where several development teams work in parallel in different platforms. That way the pilots will have
an integrated platform earlier in the lifecycle so that end user experience (and therefore end user
feedback) will be more effective.
Category: Technological, policy

2.3.2 Using QualiChain in a production environment

In order to use the QualiChain platform in a production environment, there might be a need for
changes of the rules and the legislation of each country. For instance, in Greece a new law is
needed so that the authenticity check can be done by ASEP and not by the current appointing body.
Also, there might be a need for a change in the criteria and their combination regarding the selection
staff on duty and possibly, in the future, for all other positions. Furthermore, there would be a need to
adjust the user interface and list of values choices in order to adopt to different and sometimes
conflicting business needs of different pilots. In order to maximise the utilisation of the QualiChain
functionalities in a production environment, it would be necessary to integrate the provided QualiChain
APIs with ASEP’s information system. Of course, this would require significant analysis and development
effort in coordination with organisation internal and external development teams.
Category: Legal, technological, policy

2.3.3 Data protection

A main lesson learnt during the development of QualiChain platform is about personal data protection.
Concerns were raised on how to treat personal/ private data by all types of stakeholders. As the
QualiChain implementation coincided with the adoption of the GDPR, all stakeholders were on alert
regarding data management. Relevant regulation and frameworks were constantly discussed. The main
issue to be solved is regarding the implementation of Art. 17 - Right to erasure (‘right to be

16
QualiChain D7.6
QualiChain Pilot Evaluation and Lessons Learnt

forgotten’)2, which is in contrast with blockchain’s functionality in general. One option

adopted, via the work performed under the context of WP1, is the sign of consent forms by every
involved individual (i.e., students, professors). However, QualiChain is not storing personal data on its
blockchain.
Category: Legal, policy

2.3.4 Multi-Criteria Decision Support System (MCDSS)

The Multi-Criteria Decision Support System (MCDSS) (specifically using the TOPSIS3 method), was used
to select the optimal candidates for the advertised position and the ranking was compared to the
traditional methods used so far. As a result, the MCDSS fulfilled its purpose and the need to find
the most effective candidate is covered to a great extent. This is because all the criteria are
evaluated simultaneously and a balance is sought in the development of the candidate's qualifications,
"comparing" the profile of each candidate separately with the outline of the ideal one for the advertised
position, thus proposing the most appropriate option, something that may be overlooked by the existing
system. Other methods used by QualiChain that help decision makers find the alternative that best suits
their needs are Multi-attribute Utility Method (MAUT)4, ELECTRE I5 and PROMETHEE II6.
MAUT gives exactly the same order of preference as the current scoring system of ASEP. ELECTRE I
has the disadvantage that it provides only the score between pairs of alternatives. We must use a
second method to select the best alternative. PROMETHEE II is used around the world in a wide variety
of decision scenarios, but it has the following disadvantages: 1) it is possible for a candidate to have
the best score of all, but when compared to a particular candidate he lags behind him, which is
paradoxical, and, 2) if an alternative is added or removed from the list, the score of the alternatives
also changes, so there is a possibility that their ranking will also change. So, the most preferred method
for our use case is TOPSIS.
Category: Technological

2.3.5 QualiChain integration with ASEP system and procedures

The rules and criteria that apply in various personnel selection processes of ASEP can create a variable
factor of complexity, resulting in an increased level of difficulty for incorporating them in a MCDSS
environment, in order to obtain the interim results (ranking before interviews). Flexibility in defining
many different recruitment procedures and defining the (often changing) ranking rules in each one of
them is an important enhancement. Moreover, the process of exporting the candidates list including
their qualifications and importing them to MCDSS is not automated, which means that a file must be
exported, processed, and imported, which is time consuming and prone to mistakes. Obviously there
needs to be some enhancement to the interface between the two systems, if it is to be launched in
production.
Category: Organisational, technological.

2
https://gdpr-info.eu/art-17-gdpr/
3
https://ieeexplore.ieee.org/document/9243658
4
https://www.researchgate.net/publication/226468665_The_Multi-attribute_Utility_Method
5
https://onlinelibrary.wiley.com/doi/10.1002/hfm.20301
6
https://www.sciencedirect.com/science/article/pii/S0098300411004365
17
QualiChain D7.6
QualiChain Pilot Evaluation and Lessons Learnt

2.3.6 QualiChain Platform: Corner stone with great potential

QualiChain facilitates the transition of qualification verification to a new era. The platform, although
new, achieves the goal of secure qualification validation using the undisputable reliability of blockchain
technology. That being said, the validation procedure could be easier to use with some small
improvements. For example, the issuer hash confirms the authenticity of the qualification, but it could
also provide more qualification metadata in order to accelerate the evaluation procedure.
Category: Technological

2.4 Staffing the Public Sector – The Case of Portugal

This sub-section is dedicated to the lessons learnt from the operation and execution of the pilot entitled
“Staffing the Public Sector – The Case of Portugal” that was implemented by the Administrative
Modernization Agency (AMA) and supported by INESC-ID. The following lessons learnt were collected
during two phases. The first one during the design, implementation, testing, and deployment performed
by the development team. And the second one while piloting the platform with end users. They
represent a core knowledge that can be used to offer reliable and faster implementations in the future.

2.4.1 Learning curve for the technologies related with blockchain

The development team invested a large amount of time to study and experiment the blockchain
technologies used on QualiChain. Moreover, due to the immaturity of such technologies, some issues
were found related to the stability of the solutions and to kernel changes during the initial software
development process.
Category: Technological

2.4.2 End users lack knowledge related to blockchain technology

Despite the great interest and the positive feedback revealed by the end users when approaching the
QualiChain concepts and tools, many difficulties to understand how to use them were identified. Specific
training was required to communicate the goals and to support their activities, which introduced an
overhead to the initial rollout plan.
Category: User experience, technological

2.4.3 Clear guidelines definitions provided to the end users

A step-by-step approach was presented in the guidelines that were provided to the end users. Including
the URLs to be used, the buttons to be clicked, the expected data in each field, and the expected result.
This clear definition was of key importance to the success of the end users’ activities. The guidelines
were provided since the beginning of the end users’ experiments. We consider this as a good practice
to be followed by other projects in the future.
Category: Horizontal

2.4.4 Few standards and regulations available

Due to the innovative nature of the technological solutions used by QualiChain, there are a few
standards and regulations available to support the architectural decisions. Postponing the core
architectural decisions has a negative impact on the stability of a project plan development. We consider

18
QualiChain D7.6
QualiChain Pilot Evaluation and Lessons Learnt

that investing more focus and effort to establish standards and regulations can have a positive outcome
in future projects.
Category: Horizontal

2.4.5 Lack of integration with eIDAS authentication services

By the time we developed this project, not all QualiChain partners’ countries had notified eIDAS eIDs
(namely Greece) and there was a lack of persistence on Germany’s notified eIDAS eIDs. That prevented
the QualiChain project to leverage on eIDAS eIDs, with strong authentication mechanisms. Therefore,
QualiChain resorted to username/password and to integration with SEAL for Education credentials. This
fact created barriers to using these services in HEI portals and by employers’ HR departments.
Category: Horizontal

2.4.6 The UI could be more intuitive and user-friendly

Providing for an intuitive user-interface makes step by step instructions unnecessary. It also makes its
added value more perceptible and attractive to general users and to other ongoing projects with whom
QualiChain could cross-pollinate, namely major European initiatives such as the Single Digital Gateway
and Once-Only Technical System. The resolution of such shortcomings could further improve
QualiChain and increase the added-value that it offers.
Category: Vertical

2.4.7 Diploma verification could be more automatic

This pilot would have benefited from being able to “read” the diploma and automatically get its data
along with an authenticity and integrity check. While the degree verification functionality is successful
in validating the authenticity of a degree or qualification, a greater degree of automation would make
it even more desirable by HEIs and public administrations. Of course, the innovative nature of the
technologies used in QualiChain is often counter-productive to such additions and improvements.
Category: Vertical

2.5 Provision of Recruitment and Competency Management

Services
This sub-section is dedicated to the lessons learnt from the operation and execution of the pilot entitled
“Provision of Recruitment and Competency Management Services” that was implemented by SRS and
KBZ in Portugal.

2.5.1 Introduction of a human resources platform in public health administrative

procedures
In the human resources sector, some administrative procedures have been resisting the application of
new technologies, with many still being implemented using various traditional formats, often involving
paper documentation, and therefore resulting in extremely time-consuming and fraud-vulnerable
manual processes. Thus, certification is an essential way to validate and recognise the
achievements of professionals. In the recruitment process, the selection of candidates is drawn up
on the basis of correspondence in requirements established in the job offer and the individual skills
demonstrated. In fact, it can be assumed that the recruitment of professionals is a long process

19
QualiChain D7.6
QualiChain Pilot Evaluation and Lessons Learnt

associated with the verification of curricula of a large number of candidates, whose qualifications and
academic degrees must be confirmed and validated on a case-by-case basis. Through the technological
solution presented by the QualiChain platform, SRS validated the digital platform through its application
in health HR administrative procedures, specifically in recruitment associated with internal mobility
processes (mobility in category and inter-career mobility) and in the evaluation of the public workers
competency management, thus introducing new management systems for administrative procedures.
From a technological point of view, the QualiChain piloting and its conciliation with the procedures in
use in public administration was somehow demanding, since public administration needs to follow very
strict procedures and the actual screening process needs to comply with the individual laws of each
country, that sometimes do not have the flexibility of a private company. This made it impossible to
perform any piloting activities with the QualiChain platform in an external recruitment open to any
worker besides the public workers, since the National Legislation does not allow the publicity or
dissemination of public offers beyond the designated mechanisms. Additionally, the labour legislation
associated with career progression also proved to be somewhat limiting, as each particular career
presents different requirements in terms of the skills needed for an efficient skills management. Making
it necessary to create and develop a set of very specific competencies in order for them to be validated
in the assessment of employees' competencies by the supervisors. Nevertheless, in terms of keeping
track of internal mobility and reallocation (desires and needs), QualiChain is a big step forward that can
be easily adopted by the market and that can provide a decrease in the time needed to perform
activities that tend to be time-consuming in human resources. In addition, benefits may also
be achieved at the level of skills management by facilitating access to the competencies of public
workers, since QualiChain allows to aggregate in a single place a set of updated information necessary
for the development and career progression of public workers.
Overall, SRS believes that the results are promising and show that a platform of this nature would be
a good auxiliary tool in public human resources procedures. However, it is necessary to particularise
and frame the specific needs of public institutions and their specificities.
Category: Multiple categories, e.g., organisational, legal, technological, policy, and more.

20
QualiChain D7.6
QualiChain Pilot Evaluation and Lessons Learnt

3 Methodological Adoption Guidelines for the Utilisation of the

QualiChain Platform
The intention of this section is to create a set of guidelines that should be followed by the QualiChain
environment to ensure a strong and lasting adoption by potential users and society. Evidently, this is,
at least to a certain point, hypothetical, as it tries to predict the future and provide guidelines for it. We
have seen during the three years of the development of QualiChain that a continuous cycle of design –
implement – feedback is more suited to this approach, and, as well, this section may need to be revised
over time by the future QualiChain team. This is also why it is qualitative in nature. It is difficult to
estimate which part of the guidelines will have which influence in the future adoption and how the
interaction of the different parts will result in increased effectiveness. Another important point to keep
in mind is that learning from the past and from user experiences will not give the complete picture of
the future, remembering the famous quote from automobile innovator Henry Ford: “If I had asked
people what they wanted, they would have said faster horses.” All guidance in this section therefore
should be taken with a grain of salt.
Analysing the reported lessons learned in the pilots from Section 2, and looking also into the reported
results of the KPI analysis and the questionnaires in D7.5 QualiChain Pilots Execution Documentation,
Final Release [1], it seems reasonable to classify the different experiences into 4 classes:
• Strong points of QualiChain
• Weak points of QualiChain
• Integration with external systems
• External environment
In the following sections we present the corresponding findings and the guidelines we derive from
them.

3.1 Strong points of QualiChain

Based on the pilot results, the following strong points of QualiChain could be found:
• QualiChain provides ePortfolios where formal and informal digital qualifications are collected
which can easily be verified.
• Support in building lifelong learning pathways by providing in a single place an overview of the
development of the user’s career, analysis of missing skills and recommendation of additional
courses.
• Overview of the required skills in the job market.
• Support for micro credentials and smart badges that allow certification of hard and soft skills.
• Support for certification of soft skills.
• Support for informed decision making.
• Data sovereignty and personal data protection.
• Providing innovative solutions.
These strong points are characteristics that can be used for marketing campaigns of the QualiChain
environment, as they resonate well with the requirements of potential users. The further development
of QualiChain should build upon these distinctive and often innovative features and keep a strong focus
on them.

21
QualiChain D7.6
QualiChain Pilot Evaluation and Lessons Learnt

3.2 Weak points of QualiChain

Based on the feedback from the pilots, we gathered the following weak points of QualiChain:
• No support for detection of obsolete skills.
• Weak knowledge about the social aspects of increased mutual certification of soft skills.
• Innovative tools like the MCDSS require additional knowledge.
• Users report difficulties with parts of the interface.
• Weak use of linked data in interface and digital credentials.
• QualiChain data (credentials, course information, job information) should include more meta
data to be more useful for the user.
If a system has weak points that are affecting the user experience, three possibilities exist:
1. Remove the functionality that is responsible for the weak point.
2. Improve the weak point with additional effort (mitigation strategy).
3. Redesign the functionality so that the weak point is not there anymore.
It depends on the actual situation to decide which of these three possibilities are feasible and the most
sensible solution. A redesign is usually the most elegant and functional way, but also the costliest.
Mitigation strategies that gradually improve the weak point have the advantage that they can be applied
slowly, but often they do not solve the underlying usability problem. To remove the functionality may
be the fastest and cheapest solution, but at the same time risks the future potential of the system.
Looking at the weak points discovered in QualiChain, they all appear to be not easy to solve and may
require substantial investigation efforts. In this case further research on the topic is advisable before
deciding on the actual solution strategy. Some mitigation of the weak points depending on the urgency
and impact may be advisable.

3.3 Integration with external systems

A common commentary in the pilots’ feedback results was the requirement to integrate QualiChain with
existing systems in the user’s environment and commonly used job and career platforms like for
example LinkedIn7. While QualiChain already includes such an integration with FenixEdu8, unfortunately
many more systems exist in the public sector. QualiChain is already simple to integrate into other
systems due to its open architecture and APIs, but low interest exists in other players to integrate
QualiChain into their systems while the user count of QualiChain is low. This leads to a chicken-egg
problem – QualiChain cannot grow without substantial integration into other systems, and other players
will only integrate it into their systems if QualiChain is being perceived as big player in the field.
The integration into other systems is not without risks. One of the strong points of QualiChain is the
consistent focus on data sovereignty and protection of personal data, making the user the owner of his
curriculum vitae and integrated credentials. Integration and connections to other systems need to
respect this data ownership, otherwise the integration runs the risk to both jeopardise the strong point
for QualiChain and to breach the data protection laws of the European Union. While it may look exciting
to be able to export one’s credentials and CV with the click of a button into a popular private career

7
https://www.linkedin.com
8
https://fenixedu.org
22
QualiChain D7.6
QualiChain Pilot Evaluation and Lessons Learnt

platform, handing over all access rights to one’s personal data for data mining to the company behind
the platform in the process may not be what the user desires.
In this case a two-tier effort may offer the best solution. A new market study could offer insights into
which other players in the market QualiChain could use as steppingstone to create user traction. In a
second step, an adequate form of integration or connection with QualiChain could be designed that
ensures the protection of user data sovereignty and other strong characteristics of QualiChain. By
continuing its open approach to the system architecture, open standards, protocols and open APIs,
QualiChain at the same time can ensure it continues to be both easy to connect to or integrate and
being perceived as a common standard for education credential management platforms.
QualiChain can also continue to use the approach of running pilots in selected industries or public sector
agencies, increasing integration and connectivity with commonly used software platforms there one at
a time.

3.4 External environment

Finally, QualiChain exists in a public environment that affects both the possibilities and applications of
the platform, and the perceived usefulness by its users.
The following points were evident from the pilots’ reports:
• In the public sector, as well as in Academia, common workflows are usually highly regulated
and cannot be changed or adapted easily.
• Implementation of innovative solutions into an existing system is a complicated process and
often requires working with all stakeholders.
• Blockchain is hard to grasp/ there are misconceptions/ there is a low level of blockchain
knowledge.
• Local laws enforce paper-based credentials or may even prohibit digitisation of services or
announcing job offers outside explicitly named channels.
• Innovative solutions may create fear because people do not know what to expect, as they have
no previous experience with it.
In these cases, the possibilities of QualiChain to change something by itself are very limited. Workflows
in QualiChain can be adapted to local regulation and laws, while adapting regulations and laws to new
innovative workflows is not possible inside QualiChain. To be innovative, it makes more sense to speak
to decision makers and official representatives of QualiChain users and explain the problems and missed
opportunities. This may generate public interest and may lead to the necessary changes in external
rules, laws, or workflows.
In the case of incorrect information of the public, like for example regarding the properties of
blockchain, it seems the correct approach is in educating the public. In QualiChain, the Open University
already follows this approach by running a Badged Open Course on “Decentralising Education Using
Blockchain Technology”, and all partners of the consortium are already publishing information and
engaging with the public on various levels to educate. QualiChain may need to adjust and intensify this
public education effort with additional publications oriented at the general public or specific
stakeholders.
Informing the public of the possibilities of the technologies and their implementation in QualiChain will
also help to reduce potential fears against these innovative solutions and gain trust, which again will
foster the wider application of the system.
23
QualiChain D7.6
QualiChain Pilot Evaluation and Lessons Learnt

4 Conclusions
The current document provides a comprehensive list of lessons learnt as a result of the execution and
evaluation of the QualiChain pilots and the precious feedback collected from the end-users involved.
The lessons learnt drove the composition of a set of guidelines for the future adoption of the QualiChain
solution.
QualiChain already offers many strong points and possibilities of integration with other systems, so that
it seems likely it can succeed in securing more users and applications after the end of the H2020 project.
However, it is important to continue to build on the reported strong points, while mitigating or solving
weak points, to advance the integrations and to finally continue to work for a favourable environment
to digital education credential solutions so that QualiChain can continue to add value to the life of its
users.

24
QualiChain D7.6
QualiChain Pilot Evaluation and Lessons Learnt

Annex I: References
[1] QualiChain, D7.5 QualiChain Pilots Execution Documentation, Final Release, 2022.
[2] QualiChain, D2.2 QualiChain Stakeholders’ Requirements and Use Cases, Report, 2019
[3] QualiChain, D7.1 Pilots Preparation Handbook, Report, 2019.
[4] QualiChain, D7.2 QualiChain Pilots Readiness Documentation, 2020.
[5] QualiChain, D7.3 QualiChain Pilots Execution Documentation, Release I, 2020
[6] QualiChain, D7.4 QualiChain Pilots Execution Documentation, Release II, 2021

25