Assessing Historical Thinking Skills: Scoring the AP U.S. History Document-Based Question

Uma Venkateswaran and Rick Morgan

This article discusses the ways in which historical literacy is measured through the Document-Based Question (DBQ) on the Advanced Placement Program United States History Examination. A brief discussion of the nature and purpose of the DBQ is followed by an overview of the scoring process and the steps taken to ensure accuracy and fairness in scoring standardized examinations.

Overview of the Examination

In May 2002, the AP U.S. History Exam, which tests knowledge gained in an introductory college-level survey course, was administered to more than 220,000 students nationwide. The exam is three hours and ten minutes long and consists of two parts: the multiple-choice section (80 questions and 50 percent of the composite score) and the written essay section (the DBQ and two thematic essays). The DBQ was introduced in the AP exam in 1973 with the intention of assessing not merely students' recall of historical information, but also their ability to understand and interpret primary source material (1). The DBQ is an essay question that contains a set of about nine primary source documents, including a mix of graphs, tables, visuals, cartoons, and excerpts of speeches and writings that expose the student to qualitative as well as quantitative sources. Students are required to use the documents and their knowledge of historical facts and material related to the question in constructing their essays. They perform several tasks that lie at the heart of the historian's craft including: identifying the source for each document; checking for internal consistencies (and inconsistencies) within and among documents; and establishing a context for each document and for the historical question as a whole (2).

For example, the DBQ administered in 2001 explored the Cold War during the Eisenhower period. Students were asked to identify Cold War fears of the American people and evaluate Eisenhower's success in addressing them. The documents provided included the following:

• A brief excerpt from one of Eisenhower's press conferences
  
  
• An excerpt from a John Foster Dulles speech in 1954
  
  
• A Life Magazine photograph of a family in a bomb shelter
  
  
• A Saturday Evening Post editorial on the National Highway Act
  
  
• A Herblock cartoon from the Washington Post in the 1950s on defense spending
  
  
• An excerpt from Eisenhower's speech on the National Defense Education Act in 1958
  
  
• A table of statistics on life expectancy, GNP, government spending and defense spending during the years 1949-1959.
  
  

Scoring the Examination

The DBQ is scored by college faculty and high school AP teachers who serve as readers for the AP Exam in June. To ensure that the approximately 800 readers apply the same standards to all essays, a group of experienced exam evaluators and ETS consultants read hundreds of students’ essays and then compose a scoring guide and sample packets. The scoring guide briefly outlines the main qualities of essays in four ranges (7-9, 4-7, 2-4 and below 2) on the scale of 0 to 9. Sample packets contain essays that are representative of each score point on the scale. The standards are then discussed in detail with all the readers--who also read the sample packets--so that each essay will be judged on the same basis. The Reader Management System (RMS) tracks the distribution of scores assigned by individual readers. The RMS makes it possible for the chief reader and table leaders to identify any anomalous patterns in scoring and to ensure that the scoring rubrics are applied consistently. Some of the essays evaluated by each reader are also read and scored by table leaders to ensure consistency. In 2001, for example, table leaders reread and then scored about 1 in every 60 of the DBQs that had been read and scored by the readers. The table leaders agreed with the reader's score 67 percent of the time.

To use an example from our Cold War DBQ, top-scoring essays (in the 7 to 9 range) had to have both a well-defined thesis that identified Cold War fears and a strong evaluation of the Eisenhower administration's efforts to confront those fears. In addition, they had to use a number of documents effectively and support the thesis with substantial and relevant outside information. Typically, these essays situate the documents in context and demonstrate an understanding of interconnections. For example, a top-scoring essay linked the Dulles speech with the Life Magazine picture of a bomb shelter as follows:

Perhaps the most pressing fear of America was the possibility of global armageddon brought on by a nuclear war. These concerns were exacerbated by the brinksmanship policies of Dulles . . . this fear was manifested in the building of many . . . fallout shelters showed in Life photographs (3).

The essay also included a sophisticated discussion that wove in considerable outside information, such as Alger Hiss, Ethel and Julius Rosenberg, the Hungarian Revolution, and Cuba. The entire text of this essay, other student sample responses to this DBQ, scoring guides, and commentaries by the chief reader are available on AP Central at <http://apcentral.collegeboard.com/> (4).

The average score for that DBQ was 4.0. Seven percent of the students earned scores in the upper range (7-9), while 16 percent of the essays were scored in the lowest range (0-3). Although the DBQ assesses different skills than are assessed by other parts of the exam, students who performed well on the DBQ were also likely to do well on other sections. For example, students with a DBQ score of 7 were four times more likely to receive a score in the top 25 percent of the thematic questions than students who received a DBQ score of 3.

Overall, the DBQ has proved to be an effective tool for measuring historical thinking skills—it assesses a student’s ability to use primary source materials to construct valid arguments about the past.

Rick Morgan earned his Ph.D. in quantitative psychology from the Ohio State University in 1984. For the last fifteen years he has worked on the Advanced Placement exams at ETS, first as a psychometrician and most recently as a program administrator. Uma Venkateswaran earned her Ph.D. in U.S. History from Case Western Reserve University in 1990. For the last ten years she has worked as a U.S. history content specialist on several programs including the Advanced Placement exams at ETS. "AP" and "Advanced Placement Program" are registered trademarks of The College Board.

1. Steve Kline, "The Genesis of Shorter Document-Based Essay Questions in the Advanced Placement American History Examinations," Perspectives 21 (May-June 1983): 22-24. See also, Michael S. Henry, "The Intellectual Origins and Impact of the Document-Based Question," Perspectives (February 1986): 14-16.

2. Luther Spoehr and Alan Fraker, Doing the DBQ, Teaching and Learning With the Document-Based Question (The College Board, 1995), 3.

3. AP U.S. History Released Exam (The College Board, 2001), 53. This publication contains actual multiple-choice and free-response questions from the 2001 exam, student responses, and Chief Reader commentary. It can be ordered online in the College Board Store at <http://store.collegeboard.com/>.

4. All of the free-response questions, scoring guides, student sample essays, and scoring commentaries from the 1999, 2000, 2001, and 2002 AP U.S. History Exams are published on AP Central <http://apcentral.collegeboard.com>. On the AP Central home page, select the left-of-page menu button, "The Exams," and "Exam Questions" in the breakout menu.