ABET Program Certification
HCI 551 Project 2

download Project Documentation (.pdf)

The following sections discuss the results of the implementation evaluation. Analysis of the interaction paradigm from phase one will be outlined, as well as, the advantages and pitfalls of the interface alternatives from the second phase.

The purpose of this phase of evaluation was to assess the schema of interaction between the different information items: ABET Performance Indicators, Software Engineering Program Objectives, Courses, and Students, due to these items existing within a complex, circular interaction graph. The schema was proposed by the project stakeholder, i.e. the professors of the SUNY Oswego Computer Science department who will be tracking the ABET certification process of the Software Engineering major.

Student participants engaged in a semi-structured review of a static HTML representation of the interaction, consisting of four HTML tables on distinct HTML pages that display the interaction of items. As was expected, the participants noted that the manipulation of the information items, such as modifying mapping between items, or adding, removing, or altering information items required a direct manipulation of the HTML. It was also noted that this is an error-prone process, as it is likely that not only errors will be made when modifying the items (as no context-sensitive restriction can be provided), but also introduces the possibility of making an error when altering the HTML code. This was declared to be a tedious and cumbersome process. Furthermore, navigation between the HTML tables must be inserted manually by altering the HTML, which is also error-prone and cumbersome. Five out of six participants expressed concerns with the presentation by static HTML and one participant even mentioned the need for a dynamic visualization approach and suggested the use of a form-like interface. This suggestion has been suggested to other participants if they didn't recommend it by themselves and all of them agreed that this would be an appropriate method. The participants were naïve to this study; therefore, this outcome is very fortunate, as it was the proposed goal of the second phase.

On the positive side, all participants were able to grasp the interaction scheme of the information items with only minimal explanation. The participants were only briefed on the certification process for an IT major to receive ABET certification, and, by reviewing the way the information was represented on the screen, were able to conclude the interactions correctly. The majority of questions regarding the ABET certification process and interaction of Performance Indicators, Program Objectives, Courses, and Students were also able to be answered correctly. It can hence be concluded, that the interaction schema that was proposed by the project stakeholders is successful in modeling and visualizing the interaction of information items and thereby simplify the interaction graph.

The appropriateness of the interaction schema was established in phase one, this prototype tries to visualize the information items in a form-like manner in accordance with the recommendation and agreement of the participants in phase one. When presented with the applet prototype, participants appealed to the cleanliness and form-like appearance of the interface. All participants understood that different panels represented different views of similar data, however with slightly different, more specific focus on each individual tab. A generally well-understood metaphor seemed to be the use of pull-down menus and list boxes as filter elements that allow the current view in the panel underneath the filter element to be ordered accordingly. Only one participant mentioned that the words “filter by” should be included to make it more obvious as to the purpose of the pull-down menus. A very positive aspect about the interface was the fact that the adding and modification of information items is done in a central location on a dedicated panel. Participants appreciated the cleverness of this approach, as items can be reviewed in all other panels, and quickly switch into “modification mode” by moving to the dedicated modification tab on the interface.

Another successful interface metaphor is the arrangement of detail tabs that display the mapping of different information items. The panels are aligned such that the more general information items, like Performance Indicators (i.e. information items with rather vague descriptions), are located farther to the left, whereas more specific information items (i.e. specific assessments of students' performances) are farther to the right. This arrangement therefore follows the natural left-to-right orientation that one would expect from text representations, but also introduces a hierarchy of the items according to their specificity.

A negative point about the Applet prototype was the missing of a context-sensitive help. All participants consistently asked for tool-tips or a help menu. Furthermore, the interface was considered not verbose enough. Feedback is necessary to explain what action was currently performed by the user (which will increase user confidence in the system) and what items can be expected to represent what. These are only superficial problems that can be addressed by simply adjusting the labeling of buttons, panels and tabs, or adding tool-tips to existing buttons and adding a status or information bar; this prototype can be considered a success.

After the participants reviewed the GWT prototype, most of them indicated that the applet prototype should be equipped with a user authentication feature that, depending on the permission level of the authenticated user, allows either to only view or to also modify items.

The GWT prototype also benefits from the positive evaluation of the interaction schema in phase one. This prototype also employs tabs to show relational data in a stream-lined fashion. While an overview screen allows for a quick glimpse of the existing items, participants critiqued the need for a “Details” panel to show the relationship of one specific item to another. It is not possible to review a collection of items and their individual relationships to other items at the same time. This was considered a mission-critical pitfall of the GWT prototype. Another issue of the GWT prototype is that – similarly to the applet prototype – tool-tips are missing from buttons, drop-down lists, radio buttons, and list boxes. This, however, is a problem with the GWT technology, as it is not possible to add such a feature to control elements of the interface. The final point of criticism is the color scheme that was used in the HTML wrapper. The prototype is fully customizable using Cascading Style Sheets however, for the purpose of this evaluation, the default color scheme was employed. This color scheme displays black text on a brown background, which was frequently experienced as distracting and hard to read due to the low contrast. One user, though, pointed out a possible usefulness of this effect: while “Detail” dialogs are modal in nature (i.e. does not allow the parent panel to be manipulated while the dialog is showing), GWT does immediately effectively communicate the modality of the dialog to the user. Hence, the color scheme could help in more effectively communicating the modality of a dialog.

A successful aspect of the GWT prototype is the Login Screen feature that comes with the ability to display certain editing-related control elements on the interface, and thereby allowing users, depending on their permission level, to modify or review the information. As the prototype automatically removes elements used to add, delete, or modify information items, users cannot be tempted to attempt an action that is not granted to them. An alternate approach pointed out by two of the participants is that a button could be “grayed out” to symbolize that this control is currently not active.

Another positive aspect of this prototype is that it is more aesthetically appealing than the Java applet. Participants appreciated the elegant design of control elements and the clean representation of information items. However, since GWT was designed to ease the development of elegant, dynamic, data driven web applications, this is not surprising.