Study Break

This week is study break followed by the university wide mid-semester break. Time to relax after the stressful submissions of assignments and prepare for the submission of next assignments due after the break.

No Lecture – Practice Evaluation of Prototypes in Class

There is no lecture this week but we will ask our fellow designers/developers to obtain constructive feedback to improve the proposed product. I have prepared a questionnaire of open-ended questions to evaluate the functionality and performance of the proposed design of Transmin HMI.

Each one of us, gave constructive feedback on others design for the proposed product. This feedback will be evaluated to prepare an evaluation report to be submitted as the second assignment.

Participatory heuristic evaluation.

Participatory Heuristic Evaluation (PHE) is an extension of the traditional Heuristic Evaluation where a number of design guidelines are applied to a design or prototype by usability experts. PHE uses the same techniques, however users are included as ‘work-domain expert inspectors’. Extra heuristics are added to include the user experience. In addition to the 13 heuristics identified in heuristic evaluation, Participatory Heuristic Evaluation facilitates the checking of

  •     task flow
  •     suitability of design to task
  •     suitability of design to user
  •     quality of work produced



Muller, M. J., MAtheson, L., Page, C., & Gallup, R. (1998). Participatory heuristic evaluation. Interactions, 5, 13‐18.

Lecture – Evaluation (Theory)

Third stage of product design: Evaluation of prototype by users.

Iteratively developing prototypes and performing usability testing allows to build usable products and applications. These activities identify potential problems, allowing correction before launching the final product or design.

Evaluation is  the proper understanding of the usability and enjoyability of a product design, involving a specified group of users that perform specific activities or tasks in a specified environment or work context. Evaluation identifies the flaws in the design, provides users’ views of the design and helps in taking informed decisions to create a user-centred design. Different aspects of the design can be tested like the functionality, the aesthetics, the safety, the learnability, the memorability and the intuitiveness. The evaluation techniques depend on the purpose of the design or product and its users.

Evaluations are conducted in three different settings:

  1. Controlled settings: Users’ activities are controlled and done in labs using usability testing and controlled experiments.
  2. Natural settings: There is no control over users’ activities and this involves field studies in public places, user homes or online communities.
  3. Setting not involving users: Experts, researchers or consultants analyse the different aspects of the interface through inspection methods, heuristic evaluation or cognitive walkthroughs.

We need to consider the following during evaluation:

  • goals
  • questions for evaluation – what we are evaluating – functionality, aesthetics, etc.
  • evaluation method or approach
  • practical issues and drawbacks
  • ethical issues
  • evaluation, analysis, interpretation and representation of the data.

Evaluation is of three types:

  1. Formative evaluation: This is an ongoing process carried out during the design stage of the product lifecycle by internal teams or external experts. It is done in early stages of design to predict the usability of the product, check the identified users’s requirements by seeing the use of an already existing system in the field and to test ideas quickly. In the later stages of the design process, this is done more to identifies the issues faced by users and improving the product.
  2. Summative evaluation: This type of evaluation assesses the value or worthiness of the product and checks if the product meets the objectives. It also considers any new information arising after the start of the product development and the impact of formative evaluation on the product.
  3. Impact evaluation: This is done three to six months after the implementation of the design to check the effectiveness in changing times. It is mostly used to support the application of interactive media materials.

Iteratively developing prototypes and performing usability testing allows to build usable products and applications. These activities identify potential problems, allowing correction before launching the final product or design.

During an evaluation process, it is important to consider the users’ characteristics, the activities or tasks they perform, the environment of the study and the nature of the product being evaluated. To perform evaluation certain tools are required, four commonly used evaluation tools are:

  1. Observation and monitoring: Direct observation to understand how users interact with a product in their natural settings. Indirect observation like video recording or remotely observing users.
  2. Collecting users’ opinions: Sturcture, Semi-structured or Open-ended Interviews to find out what users’ think about the product. Questionnaires and surveys with closed and open-ended questions to reach a large number of users.
  3. Interpreting situated events: Interpretive evaluation helps the designers to understand how users use the product in their natural environment and the effects of their surroundings on different tasks they perform.
  4. Predicting usability: Predictive evaluation helps to anticipate the problems users’ face when using a product without actually testing the product with users. This can be done by psychological modelling technique such as keystroke analysis or by getting experts to review the design and predict the problems that a typical user might encounter. This technique requires specifications, mock-ups or prototypes.

A thorough understanding of the evaluation process was provided, this helped me to plan and develop an appropriate evaluation tool for my project. My evaluation tool was a structured questionnaire which evaluated the aesthetics, the flow, the easy identification of elements of my design.

One for all and all for one?: case studies of using prototypes in commercial projects.

Bryan‐Kinns, N., & Hamilton, F. (2002), discuss about the importance of communication and collaboration between usability experts and the potential users.  They have presented case studies that highlight the importance of representations in communication between not just usability experts and end users, but also graphic designers, clients, and technologists.  These case studies illustrate the need to select appropriate representations for the target audience and the stage of system development. There is a need to identify the relationships between representation fidelity, target audience, and stage of development; which can be used to take informed decisions about the appropriate representations.

They argue that the development of commercial products involves competitive teams wherein members of the team perform parallel and inter-dependent activities. That is why design in commercial product development context is a highly social process in which  prototypes’ representations are at the core of stakeholder communication and the coordination of activities. They put forward two case studies that clearly demonstrated the instances of communication and coordination breakdowns when one prototype is used to support multiple activities involving multiple stakeholders. Project C involved multiple representations of the prototype and showed better prospects supporting the stakeholders. Thus, they concluded that one prototype does not support all the communication and activities in complex projects. They proposed that the breakdowns occurred because different activities required focus on different features of the prototype. So, when one prototype is used, different users will have to adjust the representation to suit their needs which could be anything from a physical transformation of the prototype (e.g. the creation of use cases) or the cognitive transformation of the prototype (e.g. users changing their focus to other attributes).  These changes should be anticipated in the planning stage to avoid impact on the project deadline and budget. Organisations should consider the users or stakeholders and the activities or tasks they undertake, to guide them to produce appropriate prototypes. They suggested that further research to clarify how the dimensions in their framework interact and the factors that modulate that interaction.

A very clear explanation why prototypes need to be designed based on the context, the tasks involved and the users.

Bryan‐Kinns, N., & Hamilton, F. (2002). One for all and all for one?: case studies of using prototypes in commercial projects. Paper presented at the NordiCHI ’02 Proceedings of the second Nordic conference on Human‐computer interaction.

The anatomy of prototypes: Prototypes as filters, prototypes as manifestations of design ideas.

This is a really good article and gives a good understanding about prototypes and their actual purpose. It elaborates on some of the ideas about “mixed-fidelity” to contrast two views of prototypes:

  • To test ways of satisfying requirements
  • To systematically “traverse a design space”

The article proposes that prototypes should differ on the design dimensions in order to FILTER the (actual or hypothetical) design space to focus on particular regions. Furthermore, the MANIFESTATION of a prototype should be as simple or efficient as possible as long as long as it performs acceptably in its filtering role.

It differentiates between prototypes, which are a representation of the design idea, and prototyping which is the activity of using prototypes to facilitate design. According to the authors, prototyping provides a means to traverse the design space and allows for the creation of meaningful knowledge about the envisioned final design.

It specifies that what you focus on in a prototype (which they term ‘filtering’) and the materials you use to achieve it (the ‘manifestations’) play a key role in how the user will interpret the design, and without careful consideration to these effects there is a high probability of obtaining unintended user feedback. For example, paper-prototyping often cannot effectively communicate complex and detailed levels of interactivity. However they can facilitate critical feedback from users since they are very obviously not finished products, and therefore users are more willing to make suggestions for change. Furthermore, they assert that the materials used in a prototype play a key role in how the user will interpret the design and interact with it.

The fidelity (from the material, scope and resolution) of the prototype is an important factor in how users and others interpret the design, and therefore the implementation of prototypes with respect to these factors needs to be carefully considered.

Lim, Y‐K., Stolterman, E., & Tenenberg, J. (2008). The anatomy of prototypes: Prototypes as filters, prototypes as manifestations of design ideas. ACM Transactions on Computer‐Human Interaction (TOCHI), 15(2), 7:1 ‐ 7:27.

Lecture – Prototyping

We are in the second phase of the interaction design: prototyping. The simplified iterative cycle in design is: Establish requirements –> Design Prototype –> Evaluate the prototype with users –> Identify the problems –> Redesign –> Re-evaluate  and go on till the final product is ready.

A prototype is an early sample or model built to test a concept or process or to act as a thing to be replicated or learned from. A prototype is designed to test and trial a new design to enhance precision by system analysts and users. Prototyping serves to provide specifications for a real, working system rather than a theoretical one.” (Prototyping Definition. PC Magazine. Retrieved 2012-05-03)

A prototype tries to illustrate and validate the visual design, effects, interactions, and use of data. Although, it is limited in its features and functionality it has the following advantages:

  • helps the users to test the design ideas and concepts and gives users an idea of what the final system will look like
  • cost effective as the development costs are reduced
  • assists to identify any problems with the efficacy of earlier design, requirement analysis and coding activities
  • helps to refine the potential risks associated with the delivery of the system being developed
  • helps to deliver the product in quality easily
  • user interaction available during development cycle of prototype
  • makes patenting easier.

Prototypes are of mainly two types:

  1. Low fidelity: Storyboarding, Sketching, Chauffeured prototyping and Wizard of Oz prototyping are some of the low fidelity prototype techniques.
  2. High fidelity: Full prototype, Horizontal prototype, Vertical prototype and High fidelity prototype are some of the techniques.

It is advisable to use different kinds of prototypes in different phases of the iterative cycle as this give a two-phase view of the design and reduces the risks of possible flaws.

Generally,Requirements animations is used in the initial phase of the prototyping to demonstrate the possible requirements in a prototype. Another alternative is Rapid prototyping which is used in collecting information on requirements and the adequateness of the proposed design. Incremental prototyping is based on the overall design in which each section is built one at a time and tested. Evolutionary prototyping is a continuous process as it occurs all through the production process and involves changes during and after development.

A screen design prototype is a very good option as it provides the visual elements, the layouts and the flow of the design, clearly giving an idea of the proposed final product and its aesthetics.

Overall, complete session on prototyping in the class involving an class activity which helped me identify the possible problems or mistakes that can take place in developing a prototype. I have selected a screen design prototype for my project, which will clearly provide the visual elements like colour, texture, font, etc. and the layout of the HMI design.