Part II: Collecting and Using Qualitative Data

Part II: Collecting and Using Qualitative Data

Qualitative data are data that describe, explain, and characterize the subject of investigation using words rather than numbers. Quantitative data collection and analysis techniques may be used, for example, to determine how often, on average, network users connect to the network, while qualitative data collection and analysis techniques may be used to characterize their use of the network in terms of what they do on the network, how they do it, and why they use the network.

Qualitative techniques are especially appropriate for use in situations where the research problem and the research setting are not well understood. When the problem and setting are well understood, the investigator can obtain information to develop survey questions and quantitative measures. When it is not clear what questions should be asked or what should be measured, a qualitative approach will be more useful.

Evaluation of academic networking is in the very early stages. Thus qualitative techniques will be especially useful in developing an understanding of the users of networks as well as the benefits and problems associated with network use. A key advantage to using qualitative data collection techniques is that participants often are interviewed and observed in their natural settings. Thus, they can more conveniently participate, they may be able to more accurately answer questions about their settings, and the researcher gets a firsthand look at the settings as the participants describe them.

Qualitative approaches allow the participants to raise topics and issues which the evaluator did not anticipate and which might be critical to the investigation. They also allow participants to express their feelings and offer their perspectives in their own words. Often participants will provide examples or anecdotes that illustrate a particular point-of-view. These anecdotes can be very powerful and persuasive when evaluators are reporting findings. This is in direct contrast to a survey in which respondents may answer only the questions which are included on the survey and, when the questions are close-ended, respondents must select an answer from a list of possible answers. In sum, the strength of qualitative research is that it is best for exploratory and descriptive analyses which stress the importance of context, setting, and subjects' frames of reference (Marshall & Rossman, 1994).

While either qualitative or quantitative methods can be used alone to assess academic networking, a more powerful approach is to combine qualitative and quantitative methods. A well designed evaluation of a network is likely to include both types of research methods. Quantitative research techniques and data collection provides a sound basis for statistical projection. Qualitative research findings should not be used to generalize to populations that are presumed to be similar to the one under study.

Overview of Qualitative Techniques

A complete discussion of qualitative evaluation research designs and social science research methods is beyond the scope of this manual. Specific information on these topics can be found in a number of useful texts, including Rossi and Freeman (1993), Babbie (1992), Denzin and Lincoln (1994), and Marshall and Rossman (1994). This section highlights a number of data collection techniques useful in developing qualitative evaluations of institutional networks. Specifics for using these techniques can be found in textbooks listed above or in other research methods texts. The selection of one or more of these techniques will be a function of the institutional resources available for data collection and analysis as well as the type of data and results desired. Potential users of these techniques are advised to consult trained individuals at their institutions for guidance and assistance.

Network Benchmarking

Benchmarking is a business measurement strategy which Terplan (1995) defines as the ". . . in-depth comparison of network management functions and instruments of two or more companies in order to establish quantifiable indicators of network management efficiency" (p. back cover). It can be used to expose the discrepancies between measures of one's own network performance and management and measures of an entire industry. Comparisons can be made against industry averages or best practices. For example, Charles Bowsher, Comptroller General of the United States, proposed a plan to reengineer various offices of the U.S. federal government. In his proposal, Bowsher (1994) recommends benchmarking government information management practices against leading organizations selected according to objective data or recognized criteria.

The first step in benchmarking is identifying what activities or processes need to be benchmarked. Terplan (1995) provides an exhaustive list of possible benchmarking indicators including generic indicators, organizational indicators, specific network management process indicators, and cost indicators. Boxwell (1994) advises his readers to choose activities that explicitly increase the organization's value.

The second step involves identifying comparative organizations. Ideally, a network analyst wants to know how the organization is performing compared to the industry average as well as compared to the organization's strongest competitor.

The third step consists of collecting the benchmarking data. Terplan (1995) offers a comprehensive assessment guide in which he suggests using his preliminary questionnaire to assess company background, applications, transmission facilities, networking equipment, personnel, costs, and basic network management. A more probing on-site interview questionnaire is then recommended to assess, ". . . investments and organizational structure; networking management functions and problems; the implementation of processes, protocols, and instruments; and the direction of network management" (p. 81). A final questionnaire, which is even more in-depth, is used for analyzing single network management processes and functions. These three questionnaires are then combined with observations of the client contact point, operations, and shift takeover. Lastly, Terplan recommends a comprehensive analysis of the aforementioned indicators for network management.

Data analysis is the fourth step in benchmarking. The quantitative data collected in step three are consolidated and entered into a statistical software package (e.g., SPSS, SAS, etc.). Specific analytical procedures will depend on the indicators themselves. Findings are then compared with findings from organizations identified in Step two. Recommendations to network managers are developed based on this interpretation of the data.

Although benchmarking is an effective measurement technique to help an information systems department become aware of its position as compared to industry standards, it is not without problems. Requirements for effective benchmarking include: researchers with the appropriate skills, training, and experience; and an adequate, representative sample of organizations from the appropriate industry. These requirements may be difficult to meet in the higher education field where a "culture" of ongoing evaluation is virtually nonexistent. Until more academic institutions participate regularly in rigorous benchmarking practices, this technique will be difficult to implement.

Focus Groups

This qualitative data collection technique is extremely valuable for obtaining naturalistic insights into how individuals perceive networks and network impacts (Morgan, 1993). With this technique, the evaluator identifies a particular group of individuals (usually 5-11 people) that meet certain criteria (e.g., members of the academic community). The individuals (who typically do not know each other) are brought together to discuss aspects of the topic at hand. The session typically lasts one to two hours, and is conducted in a conference room setting, with a moderator and a note-taker from the study team participating in the session (Krueger, 1994).

A focus group session differs from a group interview in that participants in a focus group are encouraged to make contributions to the discussion beyond simply answering the moderator's questions. The moderator should have a short list of questions to ask during the session, but these questions should be broad. And the moderator should allow participants to bring up related issues. This approach encourages users to share their perspectives, provides data based on the topics that users believe are most important, and allows the data collection to inform the evaluator about additional topics that might need attention but which otherwise would not have been identified.

Critical Incident Technique

To better understand users' perspectives, sometimes it is helpful to have users describe specific recent experiences or incidents related to the topic of the investigation. For example, the investigator might be especially interested in the users' experiences accessing class information via the network. The investigator can ask the users to recall and describe their most recent or memorable uses of the network for this purpose. This technique is likely to provide more details about their use of the network than if the moderator were to ask the users to simply discuss their use of the network to access class information.

The critical incident technique may be used in an interview or in a survey. Once the respondent describes a specific experience or incident, the investigator may probe (in an interview) or ask a standard list of exploratory questions (in a survey). In an interview, the evaluator has much greater flexibility in probing and following up on specific experiences or incidents than in a survey. The critical incident technique is an excellent approach for focusing a user's attention on a particular type of experience or incident and is useful for capturing the rich details of the experience or incident.

User Activity Logs

Another very useful approach is to have users maintain logs that describe: (1) the nature of their experiences regarding some related network activity, (2) the amount of time spent on that particular activity, and (3) the user's assessment of the usefulness or success of that particular activity. A user activity log can be designed to collect information on a range of network activities or it might focus on a particular one of special interest to the evaluator.

It is important that the participants maintaining the logs have a high degree of commitment and are provided with some rewards for engaging in this time-consuming data collection activity. Variations among individuals in their commitment and performance in maintaining the logs will have a negative effect on the results and may introduce bias to the data. Also, it is advisable to recruit a larger sample than necessary, in case some participants fail to complete the logs. Examples of user logs can be found in Doty, Bishop, and McClure (1992).

Typically, the evaluator will identify specific individuals to create and maintain the user logs over some period of time. The users may be segmented to obtain data from specific user types, e.g., novice users versus experienced users. And, depending on the nature of the study, the specific types of information to be collected in the logs may vary. User logs tend to be more useful if maintained over an extended time period, e.g., a month, so that patterns in use may be more easily identified by the evaluator. User logs are especially valuable in that they mirror actual behavior at the point when it occurs rather than the users' recollections of that behavior.

Network-based Data Collection

Another approach to obtaining evaluative information about users and networks is to use the network itself to interact with users and collect data. The evaluator can establish and moderate an online conference on the network about a particular topic and invite selected individuals to participate. Participants are informed that the online discussion will be used as data for the evaluation study. The moderator can play virtually no role in the conference, or he/she can take a very active role. The moderator may refrain from commenting online or may direct the discussion to certain topics and solicit opinions from individuals. The conference may be conducted during a pre-set period of time, or may continue for weeks or months, with participants contributing comments at their convenience.

An advantage of this data collection technique is that it allows participants' views to evolve and inform others as the conference proceeds. Software is available that organizes the conference into particular topics, permits individuals to send messages to one or more members of the conference, and otherwise manages the operation of the conference.

A related technique is to use the network as a means of administering a survey. For example, someone might put a short questionnaire on a newsgroup or listserv and ask for responses (to the evaluator and not the newsgroup or list). One benefit from this approach is that responses are already in electronic format. This technique, while easy to do, has a number of possible problems. First, the evaluator has no control over who will respond. Because of this, the responses may come from individuals who have a vested interest in the topic, thus biasing the sample. Second, many newsgroup and listserv subscribers object to the abundance of messages they receive and may discard electronic surveys or be annoyed at receiving them. Response rates may be so low as to invalidate the results obtained.

Interviews

One of the old standbys for data collection is an interview. Interviews can be done with individuals or with groups. The questions to be posed in the interviews can range from unstructured (little pre-determination of topics to be covered) to highly structured (complete determination of the topics to be covered). The success of this technique is largely dependent on the interviewer's skills. Goldman and McDonald (1987) offer a training guide for group interviews. Interviews have the advantage of allowing the evaluator to probe topics, which cannot be done in surveys. They have the disadvantage of requiring considerable time in organizing, conducting, and transcribing the interviews, as well as analyzing the data.

Group Process Surveys

A group process survey is a combination of a survey and an interview. With this technique, the evaluator selects a particular set of participants to examine a topic or an issue. In preparation for the meeting, the evaluator develops a set of discussion topics that are handed out to participants. During the one- to two-hour meeting, the group discusses a particular topic and each participant writes his/her view of the topic on the handout. The moderator can ask that participants write their thoughts on the topic as the discussion is in progress, after the discussion occurs and before moving on to the next topic, or both before and after the actual discussion.

There are a number of advantages to using this technique. First, and perhaps most importantly, the participants write, in their own words, their views on the topic being discussed so that the moderator does not have to reconstruct their comments from his/her notes at a later time. Second, this approach allows participants to be informed by the discussion and modify their views during the discussion. Finally, this technique usually results in a 100% response ratewhich rarely occurs with surveys, where people may skip questions, or group discussions, where some people may be unwilling to speak and others may dominate the conversations. A disadvantage is that participants may miss parts of the conversations when they are writing their views.

Site Visits

Site visits are similar to the case study approach (Yin, 1994) except that site visits are not likely to be as time consuming, are more informal, and can be less detailed. Generally, case studies have some longitudinal dimension to them since they are conducted over a period to time. A site visit generally is planned to obtain first-hand information from tours of specific facilities and services, interviews with individuals or groups, or observations of specific activities at the site. In addition, the site visit can be used to obtain reports, brochures, and examples of products or services made available at the site. An interesting aspect of site visits is the potential to directly compare and contrast different types of data collection techniques from different sources within the same site on the same topic.

With site visits, it is not always possible to predict in advance the range of data collection activities in which the researcher might engage. Clearly, some of the data collection strategies can, and should, be planned in advance of the site visit, i.e., scheduling interview times, tours, etc. Additional data collection opportunities, however, may arise as the site visit progresses. Indeed, the evaluator should be extremely conscious of opportunities to meet with individuals or groups that, perhaps, he/she could not have know about until the site visit occurred. One major benefit of site visits is the opportunity to obtain first-hand information about users or activities in a particular setting. Another benefit is the ability to evolve the data collection strategies on site, depending on the topics the evaluator determines are important to probe for obtaining additional information.

Scenario Development

An interesting, but underutilized data collection technique is scenario development. This can be done either as a group or an individual process. The basic idea with this approach is to have participants discuss "what if..." types of questions and construct scenarios, or likely series of events, that would need to occur if a particular vision or goal is to be accomplished. Scenario development is an especially useful technique for having participants consider possible future events, speculate about what key assumptions may drive the development of future events, and suggest the necessary elements for success in a particular scenario.

There are a number of methods for using scenario development as a successful data collection technique (Amara and Lipinski, 1983). One approach is to first carefully define the nature of the scenario to be explored, develop a one-page written description of an example scenario to use with the group (making sure it is pre-tested and revised before use), and identify appropriate topics and questions that need to be explored. For example, the scenario might be that a wealthy alumna gives the school money to install fiber optic cable to every desktop, allowing for individual use of immense amounts of bandwidth. Given that scenario, a number of discussion questions might be used with a group: e.g., what new services would become available, how would the new resources affect the way students prepare for classes, and so forth.

The views of group participants when discussing the implications and assumptions for a scenario can provide very useful insights into what users think might, or should, happen in the future. From the evaluator's point of view, these insights can be used to identify issues and policies that might be needed to deal with the issues. Researchers must be careful to choose the appropriate individuals to participate in scenario development. Some training may be necessary for the moderator.

Observations

Observations of users, in a range of situations, or in the use of various services/equipment, are useful in constructing a user perspective. Information gathered through observations can be used to describe users' activities in terms of what they do, how they do it, how long they spend doing it, and problems they encounter.

Observation can be either obtrusive (i.e., the user knows that he/she is being observed) or unobtrusive (i.e., the user is unaware that he/she is being observed). There are trade-offs, in terms of data quality and ease of collection, and ethical issues to consider in deciding which of these two approaches to take (Hernon & McClure, 1986).

A well developed data collection form is essential when doing observations. The evaluator must be able to easily and quickly summarize the activities observed, the length of time in which the user was engaged in that activity, and any comments the evaluator might have at the time of the observation. Also, it may be desirable to use more than one evaluator to observe an activity in order to increase the objectivity of the data. There are a number of useful texts that provide guidance for conducting observations (Epstein & Tripoldi, 1977, pp. 42-54).

Strengths and Weaknesses of Techniques

It is difficult to compare strengths and weaknesses of these techniques. However, there are a number of criteria that may be used in selecting an appropriate technique or group of techniques for use in addressing a specific research question in a designated setting. These include:

  • The degree to which the method can provide the appropriate data. For example, if you want to find out why network users are choosing to spend their time in certain ways, interviews might be more appropriate than activity logs. If you want to find out how much time they're spending on various activities, activity logs might be more appropriate.

  • The amount of time required to collect and analyze the data. For instance, interview transcripts may be more time-consuming to analyze than activity logs.

  • Costs associated with collecting and analyzing the data. This may include: the cost to users who may be collecting the data for the researchers; the cost of outside agents who may be required for specialized tasks like data entry or statistical analysis.

  • Requirements for training in the method. This applies to training for researchers as well as for users who may be asked to collect and analyze data, e.g., if they are maintaining user activity logs.

  • Degree to which users are required to be involved in the process. Some methods, such as focus groups, require a very limited commitment of time, perhaps a few hours, for those participating. Other methods, such as site visits, may require extensive preparation at the site and involvement with the researchers over a period of days.

  • Level of commitment required of participants Individuals who are required to collect and record data over a long period of time must be highly committed to collecting and recording this data accurately, completely, and consistently. The level of commitment expected of an interviewee is less than that expected of an individual who is maintaining an activity log.

  • Degree to which a representation of the entire community is required by the researcher. Interviews with selected individuals may not adequately reflect the range of perspectives in the community that can be captured in a group process survey.

    Some of the criteria listed above will be more important than others, depending on the goals of the research and the setting in which the research is being done.

    Key Networking Topics and Issues

    This section contains a series of suggested topics and issues that may be suitable for investigation via qualitative assessment techniques. The topics and issues identified below may relate to a number of networking applications. For purposes of presentation, however, they are included under the headers for which they appear to have primary importance. These topics can be used with a number of techniques described earlier in this section. For example, a critical incident technique could be employed asking the respondent to recall and describe a situation when the network significantly improved teaching.

    The topics and issues listed below are not comprehensive. Rather, they suggest possible topics that lend themselves to assessment via a qualitative approach. Those using the manual will want to add topics and refine those listed here.

    Network and Teaching

    1. Has the network changed the way you teach (Faculty perception)? If yes, how? Relevant network applications and functions might include:

      · Listservs

      · Newsgroups

      · Web pages

      · Software packages

      · Telnet sessions to remote computers

    2. Has the network improved or harmed the quality of teaching (Faculty and student perception)? If yes, how?

    3. Has faculty use of the network changed teaching (Student perception)? If yes, how?

    4. Has the network affected your workload or the distribution of time across teaching-related activities (e.g., office hours, tutoring) (Faculty perception)? If yes, how?

    5. Has use of the network affected the quality of teaching material you use in the classroom (Faculty perception)? If yes, how?

    6. Has the network affected your mentoring/advising relationships with students (Faculty perception)? If yes, how?

    7. Has the network affected your confidence as a teacher (Faculty perception)? If yes, how?

    8. Have you come to depend on the network for teaching (Faculty perception)? If yes, for what types of uses do you depend on the network for teaching?

    9. Do you use the network for teaching preparation (Faculty perception)? If yes, how?

    Network and Learning

    1. Has the professor's use of the network affected your learning or understanding (Student perception)? If yes, how?

    2. Has your use of the network affected the way you research and write papers (Student and librarian perception)? If yes, how?

      · Conceptualize topics

      · Gather cites

      · Obtain texts of articles

      · Write the paper

    3. Has student use of the network affected the quality of their papers (Faculty and student perception)? If yes, how?

      · What is quality?

      · What specific aspects of quality have been affected?

      · How is this related to use of the network?

    4. Has student use of the network affected grades (Faculty and student perception)? If yes, how?

    5. Has the network affected your confidence as a student (Student perception)? If yes, how?

    6. Have you come to depend on the network in your role as a student (Student perception)? If yes, for what do you depend on the network?

    7. Has use of the network affected your delivery of (Faculty perception) or participation in (Student perception) distance learning courses or programs? If yes, in what ways?

    8. Have specific network facilities or tools affected your learning (Student perception)? If yes, what are they?

    9. Has the availability of specific network resources affected the curriculum (Faculty and student perception)? If yes, what are these resources?

    Network and Research

    1. Has the network affected the way you research and write a paper (Faculty, Ph.D. students' perception)? If yes, how? (This might include the gathering of citations, abstracts, and full texts of articles as well as the actual writing of the paper and communication with co-authors, editors, and reviewers.)

    2. Has the network affected the way you conduct a research project (Faculty perception)? If yes, how?

      · Obtaining funding

      · Literature review (see above measure)

      · Carrying out research

      · Collaboration with colleagues

      · Publishing/publicizing results

      · Peer review

      · Dissertation review committee

    Network and Recruitment

    1. Does the network affect a potential faculty member's decision making process regarding an offer of employment (Faculty and administration perception)? If yes, how?

    2. Does the network affect a potential student's decision making process regarding an offer of admission (Student and administration perception)? If yes, how?

    3. Does the network affect institutional image and credibility (Faculty, student, and administration perception)? If yes, how?

    Network and Administrative Tasks

    1. Has the network changed the way you handle administrative tasks (Faculty and administrative staff perception)? If yes, how? Administrative tasks might include:

      · Creating class lists

      · Submitting grades

      · Planning conferences

      · Performing committee work

      · Performing tasks related to professional organizations

    2. Has the network changed the way you handle administrative tasks (Student perception)? If yes, how? Administrative tasks might include:

      · Registering for classes

      · Obtaining housing

      · Obtaining meals

      · Dealing with the bursar's office

      · Obtaining financial aid

      · Finding a part-time job

      · Finding an internship or co-op

      · Finding a full-time job upon graduating

      · Finding information about graduate programs or other educational opportunities

    3. Has the network affected the way you do your job (Administrative Staff perception)? If yes, how? Ways in which jobs may be affected might include:

      · Sharing files with other staff

      · Using online services for ordering supplies, making room reservations, arranging express deliveries

      · Communicating via email

    4. Are there administrative tasks you would like to conduct via the network but which aren't currently possible (administrative staff perception)? If yes, what are they?

    Network and Library Use

    1. Has the network affected your use of electronic information resources (Faculty, student, and librarian perception)? If yes, how?

    2. Has the network affected your ability to access information in the library (Faculty, student, and librarian perception)? If yes, how?

    3. Has the library provided public access to the network (Faculty, student, and librarian perception)?

    4. Has the network affected your ability to find the information you need in the library (Faculty, student, and librarian perception)? If yes, how?

    5. Has the network affected the types of services provided by the library (Faculty, student and librarian perception)? If yes, how?

    6. Has the library been involved in developing and providing access to campus information resources on the network (Faculty, student, and librarian perception)? If yes, how?

    Network and Help Resources

    1. Has the availability of help resources affected your use of the network (Faculty, student, and administrative staff)? If yes, which resources have affected your use and how? Help resources might include:

      · Workshops or classes on how to use a particular feature

      · Printed guides on how to use a particular feature

      · Online help available through email, newsgroups, or web sites

      · Walk-in help from a central help office

      · Telephone help from a central help office

      · Instructions from a distributed computer support person who works for your department or office

      · Help from friends

    Network and the Campus Social/Cultural Environment

    1. Has the network affected the social/cultural environment on campus (Faculty, student, and administrative staff perception)? If yes, how?

    2. Do you receive information about clubs or recreational activities via the network (Faculty, student, and administrative staff perception)?

    3. Do social or cultural organizations on campus use the network (Student, faculty, and administrative staff perception)?

    4. Has the network changed the nature or frequency of your contact with others, both on and off the campus (Student, faculty, and administrative staff perception)? If yes, how?

    Network and Professional Development

    1. Has the network affected your professional development (Faculty and administrative staff perception)? If yes, how? The following activities might be considered professional development:

      · Participation in professional organizations

      · Self-directed learning, reading, and research

      · Development of new skills

    2. How might the network be used for professional development (Faculty and administrative staff perception)?

    Network and Collaboration

    1. Has the network affected your involvement in collaborative activities (Faculty and staff perception)? If yes, how? Collaborative activities might include:

      · Projects with colleagues from other department or other institutions

      · Projects with commercial service providers and vendors

      · Projects with government agencies and other funding agencies

      · Participation with other institutions in consortia

    2. How might the network be best designed to encourage collaboration both within the institution and with other institutions?

    Strategies for Successful Evaluations

    Successful data collection and evaluation, of any type, can be enhanced with some preliminary planning and thought. While the following list of suggestions is not intended to be comprehensive, evaluators should keep these issues in mind as ways to increase the likelihood of a successful evaluation.

    Knowing Your Audience

    Prior to designing an evaluation and determining what data collection techniques will be used, it is important to recognize who the audience for the evaluation results will be. Potential audiences might be the users themselves, network managers, organizational administrators, boards of trustees, government policy makers, or others. A concern, however, is that different audiences may require different evaluation information. Thus, part of the evaluator's responsibility is to understand the information needs of the audience for whom the evaluation is being done. Some thought should be given to the measures, and thus the data collection techniques, that might be of special interest to that specific audience.

    Deciding What Exactly will be Evaluated

    Evaluators will not have the luxury of collecting and analyzing all the data they might want about a particular networked information service, due to lack of time, limited budget, inability to acquire the needed information, and a host of other reasons. Thus, they will have to define specific evaluation objectives. And, given the limitations, evaluators should focus on obtaining "actionable," results, i.e., affordable, feasible interventions or strategies which could be put in place to improve some aspect of the network.

    Developing Additional Appropriate Indicators

    Evaluators may also want to develop additional indicators specific to their networks, users, and institutions. For example, in the assessment of a part-time jobs database made available to the campus community from a particular server, it may be possible to establish the measure "number of students who found their jobs using the database." To use this performance measure, however, the evaluator will have to carefully define and operationalize key terms such as "found" and "jobs." Then, data collection techniques (selected from those listed above) would have to be considered in light of how well they would provide information on these two data elements.

    Determining Costs and Schedules

    For qualitative evaluations to be successful, they must be implemented in a timely fashion and with a clear sense of the funding required. In preparation for the evaluation, costs associated with standard budget items, e.g., personnel, supplies, travel, equipment, and contract services, should be identified. There is no use initiating an evaluation if there are inadequate resources available to complete it. It is better to complete a smaller, less costly evaluation than to attempt a large-scale evaluation and run out of funding during the process.

    Equally important is developing a schedule for the completion of the project and detailing the key tasks that will have to be done over the duration of the study. There are many types of tasking charts and project management software programs currently available that can assist an evaluator with scheduling. Such scheduling insures that everyone involved in the project knows what tasks are to be completed by whom and when. Further, the scheduling allows the evaluator to monitor the progress of the evaluation more globally and effectively and identify possible problems while they can still be resolved.

    Identifying the Appropriate Study Participants

    One problem often encountered by new evaluators is in identifying appropriate participants. For example, in the evaluation of dial-up access to the network, simply collecting data from a random sample of network users may not produce enough participants who have actually used or know about the ability to dial into the campus network. The general rule of thumb is: do not expect users to provide you with information about things that they know nothing about!

    Sometimes, obtaining the right mix of participants is critical. This concern is especially important in focus group sessions. If some members of the focus group are extremely knowledgeable about a particular network service and others are not, the group dialogue can be extremely skewed. One strategy is to carefully consider whether you need information from novice, intermediate, or expert network users. Another is to use a filter question in interviews and surveys to determine the type of user and his/her background before you proceed with collecting the information you require.

    Developing, Pretesting, and Refining Data Collection Instruments

    No data collection instrument should be administered without first being carefully developed, pretested, refined, and often pretested a second time. Qualitative data collection techniques require data collection instruments that make sense to the participants. One useful approach is to have the data collection instruments reviewed by (1) someone with experience in the data collection technique you wish to employ, and (2) a group of individuals who are members of the user group from which you will be obtaining the data.

    Administering Instruments Appropriately

    The logistics associated with administering data collection instruments can be formidable. Yet overlooking any of these details can seriously affect evaluation outcomes. Key concerns are: getting the full participation of all subjects, providing a forum for participants to discuss issues and concerns that the moderator does not bring up, and documenting participants' comments completely and accurately. To minimize these concerns, the following may be helpful:

  • Arrange for a pleasant setting to conduct the focus group, surveys, transaction logs, interviews, etc., and typically, provide some refreshments and amenities

  • Consider the order and development of topics to be discussed in the session

  • Record and analyze the content of the session while it is occurring (e.g., use note takers, audio or video recorders, etc.)

  • Manage and moderate the session in a positive and productive manner (prevent individuals from dominating or disrupting the session)

  • Send follow-up thank you notes.

    Once again, it is essential that these logistical concerns are considered and resolved within the data collection process.

    Presenting Study Results and Findings

    Evaluations incorporating qualitative data collection techniques typically fall under the heading of "action research," i.e., research that is intended to assist in the decision making process or assist in policy making. Thus, if the evaluation is to be successful, the findings have to be presented to decision makers in such a manner that (1) the decision makers are fully aware of the findings, (2) decision makers completely understand the findings, and (3) specific implications and recommendations are made explicit.

    Summary

    Users of this manual can make the most effective use of this section by combining these qualitative data collection techniques with other data collection techniques presented later in the manual. None of these techniques alone will tell the entire story, but the combination of these techniques can provide a valid, reliable, and rich picture of academic networking.

    Manual users should expect to customize their data collection efforts to their own networks, users, and institutions. Some of the techniques described here may be more useful in some situations than in others. Likewise, some of the topics and issues suggested for investigation may be more appropriate in some situations than in others. Evaluators should consider the techniques and topics described here to be a menu from which they can select items most appropriate to their environments and interests.

    To Part III: Measures