|
Among the growing community of European project participants there is a
deep understanding of the need for evaluating project results and products.
But does this understanding also include internal monitoring and project
process evaluation?
In the Interact project I had the honour of being the internal evaluator –
in other words to perform a formative analysis of the project process.
This meant that I took part in all the project meetings – In Oslo, in
London, in San Sebastian, in Härnösand and in Iasi.
There was also an external evaluator who had the task to evaluate the
learning activities and the Internet-based problem-solving role-play. The
division of tasks between these two different evaluators was very clear.
For many project promoters and coordinators it still isn’t self-evident
what to expect from an internal evaluator. Therefore it needs to be
explained that the task of the internal evaluator usually is to take on a
role of a critical friend and perform a formative evaluation through
observations of different kinds and sum these up in short reports to be
presented with remarks and maybe recommendations for the further
development of the project work. This way possible delays or wrong routes
can be avoided at early stages in projects and the outputs and other
project results can be analysed for a second qualified opinion. Because
the internal evaluator is a part of the project, has good knowledge of the
project from the inside through other own responsibilities within the
project and is thus a committed team member who wants the project to
deliver the results proposed.
What does the internal evaluation contain?
Even if the focus is on the internal project process the typical internal
evaluator observes and comments on the all the vital parts of a European
project like:
-
Aims & objectives of
the project
-
Work-plan and
timetable
-
Outputs
-
Dissemination
-
Partnership,
monitoring and reporting
-
Management of the
project
-
Observations, comments
and recommendations on these points are meant to be a help to the
co-ordinator to keep the project on track and ease the process if there
are some uncertainties or deviations from the original plans.
Evaluation plan and methods
In the case of Interact there was no detailed internal evaluation plan
proposed for the internal project process evaluation so there was a
freedom to chose road and means.
Since the notion of the roleplay and the simulation scripts at first sight
seemed rather complicated, it was important to follow up the different
partners’ understanding of the whole concept of Interact and their own
understanding of their different roles.
In short the internal evaluation structure of Interact project was very
hands-on and simple It contained:
-
Three online surveys
-
Three short
observation reports
-
Semi-structured group
discussions
-
Semi-structured
interviews with project participants by telephone of mail
-
A final internal
evaluation report
The surveys were very simple and designed to only take a few minutes to
complete but giving the opportunity for the partners to comment in free
text as much as they wanted. A simple Swedish online survey tool called
eVal was used. As the project working language was English questions and
expected comments were in this language.
The aim of the interviews of the group discussions and the interviews were
to follow up impressions from observations made during the course of the
project work in order to further any alternative suggestion for the
project work and follow up and convey any uncertainties, disagreement or
lack of confidence at earliest possible stage.
Impressions after the inception meeting in Oslo in November 2004
The first survey among the partners took place in early 2005 (mid January
to early March). The partners were asked
-
What did you expect
from the project before it started?
-
How well did you
understand the aims and the objectives of the project before the Oslo
meeting?
-
How well did you
understand your role in the project?
-
How would you describe
your understanding of the project now?
-
Are you comfortable
with your role in the project now?
-
What do you expect
from the work in the project in 2005?
Ten answers were submitted. The overall impression of the answers was that
most of the partners seemed quite confident in their roles, even if there
still were some questions-marks after the Oslo meeting. These concerns
were mainly connected to the interaction between the educational partners
and the workplace partners. The more meetings with the local workplace
partners were held the more clear the picture had become. One partner
confessed that the concept of workplace learning was completely new to him/her.
The inception meeting in Oslo was obviously a very clarifying meeting. The
participants were confident and looking forward to the first simulation
round even if some voices about lacking English proficiency and lacking
ICT skills vis à vis the Telsi platform were heard.
Observations in the autumn 2005
The first observation report was written in October 2005. It concluded
that the InterAct partnership was very active and reaching its goals as
intended and according to the anticipated timetable. In short it stated
that:
-
The Oslo meeting was
very informative and instructive
-
Partners became
gradually confident in their roles
-
There had been active
communication before the London meeting
-
Easy negotiations and
planning in London
-
The methodology of the
role-play was commonly understood
-
The simulations were
actively and successfully implemented
-
The leadership was
very involving
-
There was an allowing
attitude within the partnership
-
The committment within
the partnership was apparent
-
Responsibility taken
by all partners
-
There was a high
degree of activity
During the project meeting in San Sebastian in October 2005 there was a
short presentation about the internal evaluation and a group discussion
based on the questions concerning the project situation at that time and
looking head. Some of the remarks were as follows below.
Has Interact turned out the way you expected?
“Did not expect the team to get along so well. Relationships are so much
better than expected.”
“Amazing teamwork! Can’t do this so well at home.”
“People are patient, we take our time to get a solution that everyone can
agree upon. I did not expect to find so much room for my ideas, the
freedom of expression is unexpected. Much better than expected!”
What are the most obvious threats to success for the project?
“It is easier to offer courses. In Romania, workplaces are not seen yet as
a place for learning. Learning is for school – thereat to lifetime for the
project/model after the project has ended. The enthusiasm is there when
the money is there (generally), the workplaces are not ready (not only in
Romania)”
“Time: everyone is very busy at workplaces – go generally for training at
workplaces.”
Norway: “all workplaces are busy, the staff feels that the leaders do not
focus on training only production.”
“Amount of time out of the workplace for the participants/students”
How can you use this method in your everyday work situation/which
learning contexts?
“It opened our eyes on how new motivating and exciting it is to
communicate internationally via the Internet.”
“We can use it by building in an element of “international negotiations”
in another course.”
How avoid InterAct as a shelf product?
“Provide a platform and a certification to be put on the CV. Apply for EU
money to set up new project. Offer courses for private companies.”
“Build in elements into other courses of international negotiations and
international conferences”
“Make harder and more specific learning outcomes”
The Telsi communication platform
This platform was not as developed as many others but served the purpose
of an internal workspace for storing important project documents and
communication very well. The project management has a clear and firm
strategy of inviting all partners to discussions on topics needed for the
further development of the project.
After the San Sebastian meeting it is obvious that time has moved and left
Telsi behind. Moodle was mentioned as the most possible future tool. In
the observation report was strongly recommended to change to another
platform for the rest of the project time.
Impressions before the Härnösand meeting in January 2006
In the online survey made in connection to the project meeting in late
January 2006 the partners were asked:
After more than one whole year in the project – how do you feel about your
role in the project?
Are you happy with how the scripts that you have used have been designed?
Did you like the way the scripts were negotiated and decided?
After taking part for more that one year now – Have you used this
methodology in any other context the the InterAct project?
Compared to the methods usually used in your institution how do you regard
the learning value from the InterAct methodology of scripts and
international negotiations?
The overall impression from the 9 answers submitted was that partners were
very happy with how they had contributed to the project and they
appreciated to work in an international project. Someone says “….I am
lucky to for this project which let you give ideas anytime, from your
work-place…..” There are, however, some trouble pointed out: “Also I would
like to be more supported by my Institution in the aspect that workers (me
included) had more “free” time available for simulations and project in
general.”
The negotiations of the scripts were liked by a majority but it was
remarked “I think the first script within health and care was decided by
the project leader before the first meeting. There were many good
suggestions from members of the group, but none was chosen. At the same
time, I don’t think we had time enough to develop a script, and I really
liked it”.
It was evidently too early for the partners to implement the Interact
methodology in other contexts.
The question about the simulations contributing to higher learning values
was answered that the general notion was that it was different. But the
opinion was nevertheless that “it gives new perspectives and variation to
the methods used.” “Simply different. Depends on the target groups and
their needs and interest, it can be higher (for those in need of a general
awareness of inter-culturality, team work ICT skills etc..), or lower (for
those in need of focused language practice for ex.), if I am to compare
with our centre’s language classes” “Different and certainly generated
discussion and interest”
Observations made in connection to the final conference in Iasi in
September 2006
During the final conference in Iasi in late September 2006 there was as
short presentation of internal evaluation aspects. The following
summarizing points were commented as advice for the final valorisation
stage during the last six months:
-
Project objectives
were met and showed that Interact offers an involving method for
learning new basic skills (with its integration of ICT, language
learning, cultural exchange, negotiations etc.). It adds real value to
learning at work situations.
-
The project is a very
good example of European co-operation with a high degree of involvement
and clear division of tasks for real-time online problem solving using.
-
It is important to
have good and accepted co-operation tools and that changing a tool in
the middle of a project is not advisable.
-
Time aspects need to
be paid attention to – as it was clearly demonstrated in this project
that some of the employers hadn’t allowed sufficient time for the
simulations and that there was also a time problem for the tutors.
It was also agreed that the last stage of the project was decisive for the
dissemination of the project results. It is also decisive for a successful
uptake and implementation of the methodology and its chances of a
sustainable contribution to the diverse and rich treasure of learning
methods.
Critical friend - closing summary
Unless there are severe difficulties in a project the answer to the
question in the headline of this article “Who needs a critical friend?” is
that all the partners and the co-ordinator especially needs the critical
friend – not to criticise more than necessary, not to exaggerate
difficulties and not to challenge the partnership but to reassure the
co-ordinator and confirm and underline that a great job has been done and
that good results have been achieved. In the case of Interact, being a
critical friend was enjoyable. It was easy to conclude that the project
was a success and that workplaces will develop to be more important
learning arenas in the future. All as a result of an astonishing effort in
a good European spirit nurtured by a strong belief in the power of
education and training as an unchallenged way to develop people,
organisations and countries.
|
