This paper discusses the influence of Web-based surveys on the survey
process, the different types of Web-based surveys, and the advantages and
challenges of using Web surveys. Respondents face special challenges in
self-administered surveys, and in Web surveys in particular. This role
is examined, as well as the issues of validity, error, and non-response
in Web-based surveys. The author looks at the importance of auxiliary
languages, graphic, symbolic and numeric languages, in Web surveys, as
well as other areas of Web survey design, and concludes with
the unique aspects of Web surveys.
Web-based surveys are having a profound influence on survey methodology. “The Internet has truly democratized the survey-taking process” (Couper, 2000, Summary and Conclusion section, para.1). Survey professionals and large organizations are no longer the only people conducting surveys on the Web (Couper, 2000). Software capable of producing survey forms is available to the general public at an affordable cost, so that anyone with a Web site can conduct a survey without a lot of difficulty. For that reason, the range and quality of Web-based surveys vary considerably. The skills required to produce a Web-based survey are different from those required to construct other types of surveys. The skills for Web surveys focus more on programming ability and Web design rather than traditional survey methodology (Couper, 2001). Because of the technology involved in developing Web surveys, leadership has come from people with a background in technology, not the survey methodology professionals (Shannon, Johnson, Searcy, & Lott, 2001). “In fact, the use of Web surveys seems to have caught the survey methodology community somewhat by surprise” (Dillman & Bowker, 2001, para.3). Web design skills and computer
programming expertise play a significant role in Web surveys, and
programming errors are a problem source with Web surveys. Couper (2001)
explained how inaccuracies in computer programming of the questionnaire
affected survey results, referring to a programming error in a University
of Michigan survey that produced text boxes of different sizes. This
allowed answers of varying lengths from respondents. The importance of
code in Web survey design is demonstrated in the article by Barron and
Siepmann (1999) who included over ten pages of HTML code and JavaScript
to produce various effects in surveys used in research and teaching: questionnaires
with frames; answer columns side by side; different versions of the questionnaire
for various respondents; randomizing question order; error checking; removing
character codes from text responses; and process tracing and timing.
Web-based surveys are everywhere on the Internet. Couper (2000) stated that there is speculation Web surveys will replace traditional methods of data collection. Data that had once been collected by other survey modes is now being collected with Web surveys (Dillman & Bowker, 2001). An informal search for Web-based surveys on Yahoo by Solomon (2001) revealed over 2000 Web-based surveys in fifty-nine different categories. Not all of these were serious surveys. Surveys on the Web run the gamut from entertainment questionnaires to those with a probability-based design. Non-Probability and Probability-based surveys Couper (2000) described the various categories of Web-based surveys that he has encountered. He grouped these as either probability-based or non-probability-based. He included the following in the non-probability-based category: (1) Entertainments surveys;
Couper described several types of probability-based Web-based surveys: (1) Intercept surveys.
Differences in Presentation Dillman and Bowker
(2000) distinguish between Web-based surveys according to their methods
of presentation, either screen by screen or those that allow scrolling.
Screen by screen Web surveys allow only one question to be viewed by the
respondent at a time. The question displayed must be completed before
proceeding to the next. Other Web surveys allow horizontal or vertical
scrolling, giving the respondent the view of the entire questionnaire.
Even within these same categories of Web surveys, the surveys themselves
can differ greatly because of variations in layout and patterns of navigation
(Dillman & Bowker, 2000).
“There is no other
method of collecting survey data that offers so much potential for so little
cost as Web surveys” (Dillman, 2000, p.400). Zanutto (2001) described many
of the reasons for the popularity with Web surveys in her presentation
for her course in survey design and construction. She explained that
Web-based surveys are relatively cheap. An analysis of the cost of paper
vs. Web surveys by Schaefer (2001), for the Students Life Experiences Survey
conducted at the Illinois Institute of Technology, determined that the
average cost of paper surveys was $2.07 per student compared to the average
cost of $.88 for Web-based surveys. Zanutto described other advantages
of Web surveys as a faster response rate; easier to send reminders to participants;
easier to process data, as responses could be downloaded to a spreadsheet,
data analysis package, or a database; dynamic error checking capability;
option of putting questions in random order; the ability to make complex
skip pattern questions easier to follow; the inclusion of pop-up instructions
for selected questions; and the use of drop-down boxes. These are possibilities
that cannot be included in paper surveys. Couper (2000) saw multimedia
capability of Web surveys as a real advantage, as well as the option to
customize survey options for particular groups of respondents. It is interesting
to note that despite many of these advantages of Web surveys, Dillman,
Tortora, Conradt and Bowker (1998) found that the response rate was greater
for plain rather than fancy surveys that employed tables, graphics, and
different colors. This led Dillman et al., the authors of this study,
to question the use of fancy designs and layouts in Web questionnaires.
Web-based surveys
are not without problems. Zanutto (2001) discussed a number of issues concerning
Web surveys:
(1) Questionnaires do not look the same in different browsers and on different monitors; therefore, respondents may see different views of the same question, and, thus, receive a different visual stimulus.Although some participants in Web surveys might be concerned with privacy issues, Bosnjak and Tuten (2001) saw the meta-data that can be collected about participants, without their knowledge through cgi scripts, java applets, and user log files, as a benefit to the surveyor, and describe the questionnaire design necessary to gather this data. Thus, privacy issues become a double-edged sword, a concern to the respondent, and a benefit to the surveyor. Jeavons (n.d.) stated
that Web surveys are quite unlike other survey methods of data collection
in their execution, and this difference leads to participants acting differently
when responding to Web surveys. Using an analysis of log files, Jeavons
was able to demonstrate the number of failures, and number of repeats experienced
by respondents to Web surveys with various questions and question types,
and noticed that the first questions can cause immediate refusal or confusion
with many repeated attempts to answer it.
The role of respondents Respondents in any questionnaire are to comprehend the question, recall relevant information to answer the question, make a judgment, and select a response (Redline & Dillman, 1999). Redline and Dillman explained that in self-administered questionnaires, respondents need to perceive both the questions and the instructions. Perception, comprehension, and judgment involve not only the written language of the questionnaire, but also the other languages of the survey: numeric, symbolic, and graphic languages. Redline and Dillman called these visual elements of the questionnaire, the auxiliary languages of the survey. Respondents group the elements from these various languages together using the Law of Proximity, also known as the Gestalt Grouping Law (Redline & Dillman, 1999). This law states that people will group information in close proximity together and draw inferences from the grouping. Types of respondents Redline and Dillman (1999) cited work by Krosnick (1991) that classified survey respondents into two types: optimizers and satisficers. Optimizers devote their full attention to the completion of the survey. The satisficiers go through the motions of answering the questions, but look for ways to expend as little effort as possible doing the survey. Bornjak and Tuten (2001), through the use of metadata collected during Web-based surveys, were able to identify seven distinct response types for Wed-based surveys: (1) unit non-responders, (2) complete responders, (3) answering drop-outs, (4) lurkers, (5) lurking drop-outs, (6) item non-responders, and (7) item non-responding dropouts. This is a more detailed analysis of non-responders than the traditional categories of non-response, unit non-response, and complete response. Bornjak and Tuten described lurkers as people who viewed all the questions, but answer none. Complete responders were the survey participants who viewed all questions and answered all questions. Unit non-responders did not participate in the survey. Unit non-responders may have viewed the welcome screen and went no further, or, for technical reasons, were unable to participate. Answering dropouts provided answers to all questions viewed, but they did not view all the questions, and quit before looking at all the questions. Item non-responders viewed the entire questionnaire, but answered only some of the questions. Item non-responding dropouts were a mixture of the dropout categories. Motivation was important in determining whether someone will complete a questionnaire or drop out (Bosnjak & Tuten, 2001). Ways to Improve Response Rate Response rates for
all types of surveys have been on the decline since the 1990’s (Dillman,
Phelps, Tortora et al., 2001). Although response rates for Web surveys
are noted to be lower than mail surveys, several ways to improve response
rates have been verified in research studies. Solomon (2001) noted
that personalized e-mail cover letters, follow-up reminders by e-mail,
pre-notification of the intent of the survey, simpler formats, and plain
design have all been shown to improve response rates of Web-based surveys.
Dillman, Phelps, Tortora et al. showed that mixed mode surveys can improve
response rate as some people prefer to be surveyed in one mode as opposed
to another. For example, some people prefer responding to a mail
survey as opposed to a phone survey or Web survey. Carini, Hayek,
Kuh and Ouimet (2001) were able to show that, even after controlling for
student and school characteristics, the responses of first-year and
senior college students to traditional paper surveys and Web surveys did
not show substantial differences. Conducting surveys in two modes is costly,
but it does reduce non-response error (Dillman, Phelps, Tortora et al.,
2001. Colleting data by two different modes of communication, i.e.,
aural and visual, introduces measurement differences (Dillman, Phelps,
Tortora et al.).
“When generalized to the context of survey research, validity refers to the accuracy of the specific conclusions and inferences drawn from non-experimental data” (Satmetrix, 2001, Defining Validity section, para. 3). The white paper on validity in Web surveys, prepared by Satmetrix in 2001, cited the work of Krosnick and Chang (2001) that found Web participants’ responses “contained less random and systematic error than their telephone counterparts, as demonstrated by notably higher reliability coefficients’ (p.5). This paper offers three explanations for these differences. One is the recency effect, which can occur when questions are presented aurally, and respondents, lacking sufficient time to process all responses and place them in long-term memory, select the last response offered. A second explanation offered is social compliance in telephone interviews, where respondents tend to agree because of the presence of the interviewer. For a further explanation, Satmetrix turned to the work of Dillman et al. (2001), who explained that Web surveys are comprehended and controlled by respondents at their own pace. Satmetrix concluded that, although there were concerns and limitations with Web surveys, these limitations could be overcome when data is collected from an identifiable, known population. Sources of error The use of Web surveys has increased dramatically, but the growth hasn’t focused on survey error reduction (Dillman, 2000; Dillman & Bowker, 2001). The major sources of error in any survey include sampling, coverage, non-response, and what was actually being measured (Couper, 2000). Couper discussed how these sources of error are particularly relevant for Web surveys. Coverage is a big concern in Web surveys. Estimates of household access to the Internet vary greatly, and household access does not mean that all age groups in the household actually are Internet users. The Internet population differs from the general population in many ways, and there is great variation in Internet access between some rural and urban areas and with different ethnic groups (Couper, 2000). However, there are some communities where connectivity is almost universal. Some university campuses, for example, have universal Internet access so sample bias in Web surveys is not so much of a concern in those populations, and Web surveys are a more common survey method on university campuses that with the general population (Couper, 2000). Non-response errors are the
result from not all people in a sample willing to complete the survey,
or failing to finish the questionnaire. Generally, Web surveys have
a lower response rate than mail surveys (Couper, 2000, Solomon, 2001),
and failure to complete a questionnaire or abandonment is a major concern
in Web surveys (Couper, Traugott & Lamias, 2001). Couper, Traugott,
and Lamias hypothesized that progress indicators would improve response
rate in Web surveys where only one question appeared on a screen at a time;
however, their research results were inconclusive, as their progress indicators
took too long to load and slowed the download time for the survey.
Bosnjak and Tuten (2001) cite research that explain some of the reasons
for drop-out in Web based surveys as open-ended ended questions, questions
arranged in tables, fancy or graphically complex design, pull-down menus,
unclear instructions, and the absence of navigation aids. Dillman and Bowker
(2001) showed how survey design was related to survey error. Solomon (2001)
wondered if this lower response rate is due to our lack of knowledge of
how to increase response rate in Web surveys. Solomon described two points
in a Web survey when respondents stop completing the survey: when they
encounter a complex grid of questions and responses and when they were
asked to give e-mail address, and he noted that user logs do not show difference
in these failure to complete surveys for gender, age or education.
Jeavons (n.d.) determined that the first questions was a significant drop
out point for many Web survey respondents.
The design of a survey can affect the response rate, the dropout rate, and even the responses themselves (Couper, Traugott & Lamias,2001). Couper, Traugott , and Lamias studied the differences in response rate and responses using scrollable Web surveys and interactive Web surveys, a Web survey design that displays one question on a screen at a time. They found that if they altered the presentation of the single item screen to allow multiple items to appear on the screen, completion time for the survey was faster, there were fewer non-answered questions, and there was more similarity in answers than when questions were presented individually. They also tested the response rate with the same survey questions with radio boxes and entry check boxes, and had mixed results depending on the type of question asked. Some of their questions required addition, and they had hypothesized that the radio box version, with its dense screen layout and its horizontal format, would require more eye-hand coordination and bring response errors or greater non-response than the vertical format. Addition is normally done in vertical format, and Couper, Traugott and Lamias found that responses requiring addition were done more correctly with the vertical format and the entry check boxes. They concluded that, rather than advocating a generic design format for all Web questionnaires, i.e., radio boxes or entry check boxes; screen by screen or scrollable design, that the design should reflect the purpose of the survey, and that some designs are more suitable for some purposes or types of questions than others. Web-based surveys are in the early stages of development, and researchers still have a lot to learn about conducting effective surveys on the Web (Solomon, 2001). However, many of the same principles that govern other surveys apply to Web surveys (Shannon, Johnson, Searcy & Lott, 2001). Frary (1996) provided tips for designing quality questionnaires that are useful for Web survey design. Frary emphasized the importance of keeping the questionnaire brief and concise; getting feedback on initial list of questions with a field trial; placing confidential or personal questions at the end of the questionnaire; having response categories in progressive order, usually from lower to highest; and combining categories such as seldom and never together. Frary also provided recommendations of what to avoid: open-ended questions; the response category of “other”, that prevented respondents from selecting a provided category for a trivial reason; response scale proliferation, i.e., using a six or seven point scale when a four or five point scale would be sufficient and more distinguishable.; and asking participants to rank responses as research has shown respondents experience difficulty with ranking, especially with a list of more than six items. Web-based surveys can take advantage of the power of the visual far more than paper surveys, and the graphic nature of the Web makes the addition of graphics, color, and images quite inexpensive (Couper, 2001). Couper explained that, “The Web vastly expands the range of design opportunities and …the skills brought to the design of Web surveys focus more on programming and general Web design than on survey design” (Introduction section, para. 9). He described the wide array of response options for Web-based surveys: radio boxes, check boxes, Likert scales, drop-down menus, and skip patterns, as well as includes graphics, color and sound. Language of Survey Questionnaires Unlike interviewer administered questionnaires which are given aurally, Web-based surveys are presented in several languages: textual, graphic, and numeric (Redline & Dillman, 1999). Even though the primary form of communication in these questionnaires is textual, much of the language of the survey is visual (Couper, 2001, Redline & Dillman, 1999). The textual language of surveys includes the wording of the questions and the instructions in the responses. Couper (2001) included font size, font type, color, layout, symbols, images, animation, and other graphics as components of visual language. Even though visual language is intended to add meaning and supplement the written language, Couper observed that it can actually draw attention away from text and alter the meaning of words. Redline and Dillman (1999) distinguished between three different types of visual languages: graphic language, symbolic language, and numeric language, and emphasized the importance of these language in a Web questionnaire. Redline and Dillman referred to these as the auxiliary languages of questionnaires. Graphic language, consisting of fonts, font sizes and variations (bold, italics,) borders, and tables, helps respondents move their eyes across the page and comprehend the questionnaire. Symbolic language is sometimes used in questionnaires when arrows or other symbols are employed to help guide the respondent through the survey questions. Numeric language is frequently used in questionnaires in question numbering, and sometimes in numbering response items. Redline and Dillman pointed out that these various languages of the questionnaire work together to affect the respondents’ perception of the survey. Sometimes, these different languages of questionnaires send conflicting messages to respondents (Couper, 2001; Redline & Dillman, 1999). Redline and Dillman indicated this is particularly true with instructions for skipping questions. Respondents are accustomed to answering questions in a sequence and responding to all questions. Redline and Dillman stated that cognitive research on questionnaires suggests that respondents believe they are to answer every question. The skip instructions tell respondents to disobey what they are culturally trained to do: answer all questions and answer them in sequence. Redline, Dillman, Smiley, Carley-Baxter, and Jackson (1999) demonstrated the manipulation of these auxiliary languages by increasing font size, boldness, and arrows, as well as adjusting the placement of verbal instructions affected response rate in questionnaire with skip questions. They were able to reduce errors of commission in survey responses, those errors when a respondent is instructed to skip a question, but answers it, by more than half. Culture, too, plays an important part in the perception of a survey. It affects the way symbols and graphics are perceived, and the way different people respond to certain questions (Dillman, Caldwell & Gansemer, 2000). For example, some people are more predisposed to agree to questions than others. Acquiescence is a predisposition in some cultures, and this characteristic can affect responses, particularly in agree/disagree questions (Javeline, 1999). Even before, the widespread use of Web survey, Smith (1993) recognized the importance of the visual aspects of questionnaires. “Non-verbal aspects of surveys such as physical layout and visual presentation can also notably influence answers “ (Smith, 1993 2). Smith, who is frequently quoted by Web-based survey designers, noted how “little things” really do matter in survey design. He advised survey developers to pay close attention to physical layout and what might appear to be trivial issues, i.e., the placement of skip or filtered questions, overly-crowded design, visual and graphic images, and misalignment of response boxes. General Principles of Design Design in Web surveys is of greater importance than in other modes of surveying because of the visual emphasis of the Web and the way the survey appears in different browsers and on different screens (Couper, 2000). Couper believed that the audience and the purpose of the survey should affect the design, and that the design of a Web survey for teenagers and one for seniors might be designed quite differently. “The notion of a one-size-fits-all approach to Web survey design is premature” (p.10). Solomon (2001) noted that Web-based survey development is still in its early stages, and it yet to be seen how knowledge from other surveying techniques will be transferred to this new mode of surveying since HTML forms have their own unique design concerns. Writing from a marketing perspective, Gaddes (1998) described how to design user-friendly online surveys that would get responses. She explained that the first question was the most critical on the questionnaire and should be tied to the survey’s purpose. This advice has been given by others (Burgess, 2001, Dillman, 2000, Zanutto, 2001). Her most important advice was to “edit, edit, edit” (Gaddes, 1998, p.7). She stated that many ideas for effective online surveys came from traditional surveys: pretest questions before they go online; write an introduction for the survey which will bring cooperation from participants; use filtering questions and have questionnaires appropriate for filtered groups; divide long surveys into sections; use open-ended questions sparingly, and use incentives to get people to respond. Smith (1997), who used Web surveys as part of her doctoral research, discussed Web survey design, and offered suggestions for HTML code. She advised that long Web surveys be divided into sections, and that there should be “Clear” and “Reset” buttons for each of the survey sections, so that respondents don’t have to reset the entire survey if they want to clear one answer. She believed “Clear” and “Submit” buttons need to be in separate locations so they aren’t confused, and that word length needs to be specified for open-ended questions. Principles for Constructing Web Surveys
Dillman, Tortora and Bowker (1998) were concerned about the principles
of what they called respondent-friendly Web survey design. They described
respondent-friendly design to mean, “the construction of Web questionnaires
in a manner that increases the likelihood that sampled individuals will
respond to the survey request, and that they will do so accurately, i.e.,
by answering each question in the manner intended by the surveyor” (Criteria
for Respondent Friendly Design section, para. 8). Dillman, Tortora
and Bowker stated it is essential that Web questionnaires, like paper questionnaires,
have design features that are easy to understand, don’t take a lot of time
to comprehend, and are interesting to complete. They gave three criteria
for Web survey design and explained eleven principles of design for Web
questionnaires:
(1) Use a welcome screen that is motivating, emphasizes the ease of responding and shows respondents how to move to the next page.
(1) Include a cover letter with the questionnaire.
Dillman and Bowker (2000) turned to the literature of human computer interaction to gain insight for Web survey design. They examined the placement of response boxes and skip directions for Web surveys. Traditionally, answer boxes are placed on the left for paper questionnaires, and this practice has been transferred to Web-based questionnaires. However, there are differences in the way people work on paper and the way they work on a computer, and the Web displays questions differently than they appear on paper (Dillman & Bowker, 2000). Human computer interaction research suggests that people with greater computer skills and experience prefer right justified response boxes (Dillman & Bowker, 2000). Even though left alignment is more familiar, they hypothesized that right alignment would improve navigation, and reduce hand-eye-keyboard-mouse coordination. They were able to show that placing response boxes and skip directions on the right reduced skip-pattern errors. In addition to their work with right and left alignment of response boxes and its effect on response rate, Dillman and Bowker (2000) examined some of the other principles of design for Web surveys. They made sure Web surveys were designed for the least compliant browser, so that all respondents would have the same visual stimulus. They used the most basic HTML code with alternate code for compliance with all browsers, and they made sure that the code was validated on different browsers. Dillman, Tortora
and Bowker (1999), in their examination of the impact of principles of
Web survey design, showed that, although more Web designers have been using
more fancy features, such as HTML tables, several colors, dynamic HTML,
animation, java applets, and sound to get people to respond to surveys,
these very same features that entice people to participate may keep them
from completing the survey. Plain surveys had a higher response rate
than fancy design (Dillman, Conradt, & Bowker, 1998).
Web-based surveys are self-administered questionnaires. They are physical entities in themselves that can be manipulated, and respondents have different skills in this Web manipulation (Redline & Dillman, 1999). Navigation and flow are important in any questionnaire, but they are particularly important in Web-based surveys (Redline & Dillman, 1999). Web surveys are a visual stimulus, and respondents have control over how and even whether they read and comprehend each question (Dillman, Phelps, et al. 2001). Participants in Web surveys are less likely to take extreme positions in their responses than people that take part in a telephone survey (Satmetrix, 2001). Web surveys provide opportunities for variety in question structure, layout, and design not available in paper surveys (Couper, 2000; Couper 2001; Couper, Traugott, & Lamias, 2001; Zanutto, 2001). Redline and Baxter (1999) demonstrated there were various ways to manipulate both the verbal and the auxiliary languages of self-administered questionnaires to improve the design of skip instructions, and, in turn, improve the response rate, for skip pattern questions. Dillman, Tortora,
Conradt, and Bowker (1998) demonstrated that plain Web surveys gave a better
response rate than those with a fancy design containing colors, graphics
and tables. They gave several explanations for their findings. They
explained that longer questionnaires have lower response rate. In other
words, there seems to be a time limit that people are willing to spend
on surveys. Web surveys with a fancy design take longer to download
on slower Internet connections, thus, making them take the same time as
longer surveys. In addition, not all features of fancy surveys
may appear on old browsers or hardware. Dillman, Caldwell and Gansemer
(2000) used eleven distinguishable layout features in a survey to demonstrate
that words and graphic language combine to affect how people respond to
questions.
Web-based
surveys have had a profound influence on the survey process in a number
of ways. The survey taking process has become more democratized as
a result of Web surveys. Government organizations and big businesses are
no longer the only groups that can survey groups and collect data.
The ability to gather data through Web surveys is quite widely available.
Leadership in Web survey design is coming from people with a strong technology
background, not just the experts in survey methodology. The visual
aspect of surveys is even more important in Web surveys than with other
surveys. What was invisible in a paper survey can be made visible in a
Web survey the increased low-cost graphic capabilities of the Web. Web
surveys have reduced the cost of data collection and made data analysis
more efficient. Although there are concerns about Web surveys and many
aspects of Web survey have yet to studied, a number of researchers have
produced a body of literature that is improving the design and effectiveness
of Web survey process.
|