A Sampling Strategy for Empirical Case Law Research

Idleness of Sisyphus by Sandro Chia
October 31, 2014


Searching for trends in the common law is a Sisyphean task for several reasons. First, most cases are never reported. So, tracking trends at the trial level requires en situ data gathering. Few researchers have the resources (or stamina) to take up residence at a courthouse(s) for a long period of time. Second, data mining of reported cases runs afoul of both the licensing agreements and design of major legal databases (e.g., Lexis, Westlaw). The search algorithms of those databases are not designed to yield a reliable set of cases containing a social science variable(s) of interest. Third, even though the vast majority of cases are never reported, the remainder of reported cases is an unmanageable corpus. Great resources would be required to download, store, read, or exhaustively search reported caselaw. Fortunately, some of the collecting and “coding” work has already been done by West, and professor Sara Benesh has developed a strategy for leveraging that work.

In her doctoral research, completed under the direction of renowned empirical legal research scholar Harold J. Spaeth (e.g., of the Supreme Court database), Dr. Benesh examined judicial decision making in the federal court system. A portion of her research concerned the treatment of confessional evidence in federal appeals decisions. She wrestled with how to construct a sampling frame, or list of potential cases to review or select from. After considering a host of options, Dr. Benesh turned to the West topic-and-key number system.

The editors at West affix numerical topical markers to every reported case. For instance, Roe v. Wade is affixed with 92:k696 (Abortion and birth control), 92:k977 (Mootness), 4:157 (Health of patient; necessity), and others. Dr. Benesh realized that she could select certain key numbers related to confessions and then gather cases decided in certain years using the West Decennial Digests. While her strategy would not yield unreported cases, her approach is a reliable method that other empirical legal scholars can replicate.

The easiest way to employ Dr. Benesh’s approach is to start at the WestlawNext home screen. Click into Key Numbers and CTRL+F or scroll to the relevant legal topic, for instance Criminal Law, topic 110. Within that topic, a single key number or range of key numbers will relate to particular subtopics. Key number k31.5 is Alibi, k31.10 Abandonment; Dueling cases are covered by key number k45.30; Jurisdictional issues in criminal law fall under k83-k105. Once a key number is clicked, the sample frame is set, but an exhaustive list of cases does not necessarily follow.

Clicking into key number 59, “Principals, aiders, abettors, and accomplices in general” reveals three challenges to using the topic-and-key number system for empirical legal research. First, the subject area covered by the key number is often sufficiently large as to be subdivided. So, one must click into one of the subtopics to proceed. Second, some topics or subtopics will yield more than 10,000 results, but will be capped at 10,000 results by the search engine. Thus, researchers might need to search state and federal cases separately, or select certain jurisdictions so as to avoid the results cap. Third, even when a subtopic yields fewer than 10,000 cases for all U.S. jurisdictions, as is the case with 59(5), “Aiding abetting, or other participation in offense,” the results can be quite large; in this case 5,629. Given that large number, a researcher could select a subset of cases randomly (e.g., by using a random number generator), or by using systematic random sampling (e.g., every 9th case). If reported clearly and accurately in a methodology section, either strategy would pass muster with most social scientists.

For researchers wishing to adopt Dr. Benesh’s methodology, a final caution is in order: coverage on the West database is bounded temporally. The newest (e.g., a few days old) and older cases (e.g., >1980s) are frequently not available. Thus, researchers are advised to turn to other sources for historical or right-now research.

Dr. Benesh’s book, The U.S. Court of Appeals and the law of confessions is on reserve under “empirical legal research.”

The library holds two of her other books: Principal-agency in American courts and The Supreme Court in the American legal system (with Jeffrey A. Segal and Harold J. Spaeth).

For further resources on empirical legal research, including tutorials and links to datasets, see our Empirical Legal Research guide. For assistance designing a sampling strategy, contact Sarah Ryan, Empirical Research Librarian and Lecturer. Dr. Ryan teaches “Introduction to Empirical Legal Research” at YLS.

Published In: