Rhetorical Judgments: Using Holistic Assessment to Improve the Quality of Administrative Decisions
Roger J. Klurfeld & Steven Placek*
- Download PDF file (1005 KB) Get Adobe Reader
Federal, state, and local governments issue hundreds of thousands of administrative decisions annually. Considering the number of encounters the public has with administrative appeal agencies, administrative decisions may be the largest category of legal writing and reading interaction the public has with the legal system. Usually, appellants come to appeal organizations after they have been denied some form of benefit. Federal agencies, for example, have hearing procedures established to conduct hearings in person, by telephone, by video, or by a case review of the documentary record.1 These interactions often generate a two- to five-page legal decision that identifies issues in dispute and applies legal reasoning to the factual pattern of the case to reach a conclusion. Many of these agencies have identified writing quality — however they define it — as a priority in their strategic plans,2 but the overwhelming number of hearings and decisions, coupled with regulatory guidelines for timeliness, may subordinate this goal to other management priorities.
In ad hoc attempts to improve writing, many government agencies send writers to a slew of training programs in writing and legal rhetoric at community colleges, law schools, the National Judicial College, and other private contractor or government-run training facilities. This training is cumulatively expensive, fragmented, and often ineffective when the agency does not integrate the course writing perspective into an overall program.
Moreover, these programs rarely integrate the training with the day-to-day activities of the organization’s writers. To control the quality of written products, many government legal agencies dictate that hearing officers write decisions in boilerplate format, often contorting factual patterns and legal analysis into templates at the paragraph and sentence syntax level. Remedial instructional efforts may bludgeon writers by pointing out how previously written administrative decisions contain faulty grammatical traits, syntax errors, improper use of legal terminology, and deviation from the boilerplate. Indeed, at our agency, the National Appeals Division of the U.S. Department of Agriculture (NAD-USDA), we discovered a disturbing trend: the fragmented training approach and boilerplating parts of decisions actually caused stronger, more experienced and educated writers to write poorer decisions. These writers needed to be freed to perform under a different set of writing measures.
Improving the quality of administrative decisions at these agencies, therefore, presents a practical legal business challenge as well as the theoretical challenges embedded in the nuances of legal writing pedagogy. Amidst the pressure of issuing a large volume of decisions, agencies must contend with improving the varying skills of writers; delivering well-reasoned, clear, and reader-friendly decisions to the public; and measuring organizational performance based upon the quality of its written products.
This article proposes that a formal holistic assessment program can be an effective tool for confronting these challenges. In Part I, we describe holistic assessment and argue that adopting and emphasizing an evaluation strategy can be a powerful component of a legal writing program that results in improvements in writing across the organization. In Part II, we show the advantages of applying holistic assessment to administrative decisions. In Part III, we propose some guidelines for establishing a rubric and discuss how we adapted it to conform to the traits of legal readers and to judge the quality of legal discourse. In Part IV, we describe how to integrate holistic assessment for administrative decisions into a writing program. Part IV includes a discussion about metrics, reader protocols and training, and the use of evaluation session results by a writing program manager. Finally, in Part V, we briefly offer some suggestions about other potential applications of holistic assessment to legal writing.
I. Description of Holistic Assessment
Formal holistic evaluation has been well embedded in the writing industry for many years,3 but less so in law schools and legal writing organizations. Legal writing professors and program managers may have adopted some elements also found in holistic evaluation, such as a rubric-based evaluation, employing multiple readers for high-stakes writing tasks, or portfolio grading. To distinguish these elements from the holistic assessment method, as it is used as a systematic theoretical and practical evaluation strategy for a writing program, it is helpful to begin our discussion by describing holistic assessment.
Holistic assessment comprises a scoring method based upon a rubric of identified writing criteria applicable to the subject area.4 Raters, or readers, are encouraged to view the writing sample as more than the mere sum of its elementary parts; readers do not judge separately singular factors — such as treatment of topic, selection of rhetorical method, word choice, grammar and mechanics — that constitute a piece of writing. Rather, evaluators are asked to consider these and other factors as elements that work together; they score the writing sample on the “total impression” it makes upon the reader.
The scoring scale is usually a six-point scale, divided into two halves (a fourpoint scale is also common). Decisions that fall into the upper half — those scored four, five, or six — are satisfactory or labeled “mastery.” Lower-half decisions are unsatisfactory, or labeled “non-mastery.” Each score is described in terms important to readers. For example, a “two” might be described as “flawed writing,” while a submission that earns a “five” might demonstrate “clearly proficient writing.” After an informed reading, a rater first decides whether the writing sample is above or below the line. Based upon the pre-established holistic rubric of agreed-upon conventions, the rater then scores the essay.
To ensure statistical reliability, writing samples usually receive two or three “reads,” each by different readers. Adjacent scores (ratings that are within one point) and discrepant scores (ratings that vary by more than one point) receive thorough statistical scrutiny and inform test managers whether further reads are necessary. Training or “calibration” exercises precede evaluation sessions, giving readers a chance to apply the holistic rubric to previously scored essays and thus fostering consistency. Session leaders integrate “monitor papers,” essays with previously agreed-upon scores, into the sample pool to track reader reliability. In the end, scores can be analyzed across all writing samples by traditional data dispersion measurements. Overall judgments about good and poor writing, therefore, reflect the systemized results of reader and text interactions for specific documents assessed under controlled statistical conditions.
Holistic assessment emerged at the same time that teachers began applying new methods of teaching composition and writing to accord with modern language theory.5 Many writing disciplines have embraced it as the primary standardized formal assessment tool.6 Legal writing programs, however, can also take advantage of the attributes of holistic assessment. It privileges the reader’s role in determining writing quality, adopts an inherently judgmental disposition, and — through a rubric — weighs writing standards as they affect rhetorical and content–driven aspects of the written product. After all, legal rhetoric is intensely judgmental and self-reflective, often calling explicitly upon the audience to evaluate and weigh both the content and form of an argument. Indeed, the main purpose of much legal rhetoric is to engender a particular response in a reader or group of readers.
Some of our earliest examples of legal texts show the same kind of judgmental and rhetorical self-consciousness. For example, in introducing his defense against the charge of corrupting Athenian youth, Socrates implores the judges, in evaluating his case, to subordinate his rhetorical style to the truth of his words:
[Am I making] an unfair request [of you?] Never mind the manner, which may or may not be good; but think only of the justice of my cause, and give heed to that: let the judge decide justly and the speaker speak truly.7
The setting for the above passage is a forum, similar to all legal proceedings, that reflects a method of pleading similar to the holistic evaluation method: the speaker submits the argument before a panel; the panel represents an interpretive community; and individual members of the panel make quantifiable assessments that eventually result in a single overall judgment.8 The passage reflects a protocol very similar to the procedures we see in holistic assessment sessions.
In analyzing the passage further, we see also that Socrates addresses a cognitive element of judgment that compares to holistic assessment. As he cleaves the “manner of speech” from “truth,” Socrates exposes the millennia-old polemic in legal discourse: he asks the audience to decide what is good and true, inviting members to develop a scheme that evaluates both rhetorical effectiveness and content.9
While Socrates’ explicit invitation is one that writers and texts extend implicitly to all readers, legal readers seem to bring to the task a heightened sense of judgment. The rhetoric of law often adopts modes of discourse that emphasize objectivity and syllogistic logic, commitments that drive readers to closure.10 The emphasis on logic and closure is perhaps one reason why holistic assessment, as typically used in other disciplines, does not immediately appear suitable for legal writing evaluation. The fear may be that traditional holistic assessment may favor a form of rhetorical effectiveness over the logical and syllogistic content privileged in legal discourse. And it is true that we have found that in order to complement the judgmental disposition of legal writers and readers, holistic assessment needs some tweaking. It has to be adapted to readers confronted by legal rhetoric — whether they are judges, lawyers, professors, jurors, or students — who find themselves resolving both the effectiveness of the rhetoric and the outcome of the dispute between the parties. Holistic assessment must align the evaluator’s judgment with the disposition of traditional legal readers as they judge the rhetorical effectiveness of an argument in relation to, or as opposed to, the truth of a matter at issue.
Emphasizing an Evaluation Strategy
Some of the benefits of emphasizing an evaluation strategy for a writing program may seem obvious: formal writing assessment and measurement can become linchpins for evaluating writing quality and transferring legal writing theory and pedagogy to a plan for action and improvement. The benefit is that assessments can provide a way to measure the efficiency of the legal writing program that has been validated and statistically shown to be sound. The results of these assessments can therefore become the basis for a continuous cycle of organizational change and improvement in writing.
A formal assessment strategy promulgated throughout the writing program, however, can benefit the individual skills and cognitive processes of writers too. If implemented properly, the evaluation standards and protocols become part of the writing program interaction between reader, writer, and text. Over time, writers in the organization can serve as readers and evaluators. They read and judge the quality of their peer’s texts and base their assessments on the overall quality goals of the evaluation standards. When assessment is used properly in a writing program, writers know how their products will be evaluated, who will be the audience, and what will be the contextual conditions of the assessment. This information fosters a recursive writing process between reader, writer, and text that promotes change and growth throughout the life of a text.11 Thus, by teaching the evaluation standards and practicing assessment protocols, the holistic assessment program also supports individual writing skill development. This approach is perfectly in accord with the theoretical view that writing development is a process-oriented activity. Further, writers can bring to bear on the assessment mechanism some of the information and writing tools they have acquired from other writing courses, which adds value to previously questionable training activities.
II. Holistic Assessment of Administrative Decisions
Applied in a legal context, holistic assessment has four advantages for evaluating administrative decisions. The first advantage is that administrative decisions look very similar to analytic essays, a rhetorical form with which holistic assessment has an established history. Usually three to five pages in length, administrative decisions include an introduction, case narrative, background information, and a factual pattern. They point out the main arguments that need resolving, and consider and evaluate all sides of an issue. In coming to a reasoned conclusion, they apply regulatory criteria to a factual pattern through traditional logical and organizational patterns.
These are precisely the same elements found in analytic essays for other disciplines. For example, the analytic writing exercise for the Graduate Management Admissions Test (GMAT) requires two essays: one essay must analyze an issue question; the second essay analyzes an argument. The criteria for assessing both essays include the writer’s use of organization, logic, and analysis as well as the writer’s ability to understand and identify the complexities of an argument. Interestingly, the “issue” in legal writing compares with the analytic “issue” only in narrowing the field of topics that might be considered. Analytical essays are also common in academic areas, such as English and History, where writers summon facts and textual evidence to support or refute a thesis or proposition that is often based upon a theoretical model.
The second advantage is that holistic assessment also provides a practical business advantage for organizations that issue administrative decisions. Holistic assessment sessions can evaluate large samples of writing and produce measurable results that can be used and analyzed at higher management levels. One reason holistic assessment was implemented for the standardized Scholastic Aptitude Test (SAT), for example, was its speed and low cost in assessing large numbers of writing samples.12 In evaluating large numbers of samples, holistic assessment can produce a higher level of statistical confidence in measuring the quality of organizational writing, and at a cheaper cost than other assessment techniques. The management of holistic assessments is relatively straightforward, which makes the task less daunting for managers. Managers can put systems into place for selecting and training assessment teams and coordinating high- and low-stakes evaluation sessions. They can integrate rubric elements into performance management systems, such as appraisals and awards or bonuses.13 And since the output of these sessions is quantitative, they can integrate quality improvements into future organizational initiatives, such as training.
The holistic rubric and calibration sessions conducted before assessments provide a way to manage organizational writing quality because they provide a common language for discussing the quality of written work. Developing a common language is one of the most important aspects in increasing organizational writing quality. Every writer and reader in an organization possesses a common definition of writing quality; no longer is quality defined by what each senior partner or manager likes. Since holistic assessment provides a basis for writers and readers to judge the relationship between conventions, based on the whole document, using commonly defined and understood terms, organizational discussion about quality is more coherent, interactive, and dynamic. These discussions are very useful at our agency since we choose holistic assessment readers from the pool of writers in the organization.
The third advantage in using holistic assessment for administrative decisions is the similarity of specific disputed issues. Traditional holistic assessment evaluation exercises rely upon a prompt to which writers respond. The prompt is an important aspect for ensuring validity and reliability in holistic assessment because it controls the testing condition. Unlike a holistic testing environment, there is, of course, no universal prompt that generates a real life written work product for administrative decisions. For many administrative agencies, however, disputes center on similar patterns of disagreement about groups of regulatory criteria and language. Even when the case disputes vary, the culture of administrative decision writing creates a pattern of writing and analysis. This pattern serves a function similar to the prompt and lends itself to reliable holistic assessment results.
The final advantage for holistic assessment pertains to the wider and more diverse reading audience that administrative decisions often garner. They are written for appellants with differing educational levels who may or may not have legal representation. Poor writing signals to appellants that a given appeals system is unfair or designed to be intentionally ambiguous. Avoiding these impressions demands that authors make style (word choice, sentence and paragraph length, etc.) and clarity choices that meet the needs of both legal and non-legal readers. Also, many government appeal organizations post administrative decisions on the Internet to foster an impression of consistency, public fairness, and accountability.14 Internet publication forces writers and managers to consider how decisions will survive wider public scrutiny. Holistic evaluation easily supports an intense and varied reader awareness approach to writing quality by simulating how a community of readers might react to an administrative decision.
* © Roger J. Klurfeld & Steven Placek 2008. Roger J. Klurfeld is the Director of the National Appeals Division (NAD) at the United States Department of Agriculture (USDA). Steven Placek is a Training Specialist of the USDA-NAD, and a former Assistant Professor of English at the United States Military Academy. The views expressed in this article do not necessarily represent the views of the United States Department of Agriculture. We would like to acknowledge our colleague Jennifer Nicholson for her comments and insights on this paper.
1 Our example in the Department of Agriculture is typical. The National Appeals Division is an independent office within USDA that conducts hearings and reviews of USDA sub-agency adverse decisions, mainly the denial of benefits to farmers and firms engaged in rural development. To accomplish this function, NAD employs over 65 hearing officers located around the country, most of whom function as one-person offices and conduct hearings with authority to issue written decisions upholding or reversing the initial action. After the receipt of a hearing officer decision, appellants may also request a review from the NAD director. Typically, one-third of appellants seek review after receiving hearing officer determinations. The director’s decisions are drafted by a group of nine review officers. In all, NAD issues over 2,500 written decisions annually.
2 For examples, see USDA-NAD, Strategic Plan, http://www.nad.usda.gov/about_stratplan .html (last updated Jan. 18, 2007) and James P. Terry, Fiscal Year 2006 Report of the Chairman, Board of Veteran’s Appeal 5, http://www.va.gov/Vetapp/ChairRpt/BVA2006AR.pdf (Jan. 10, 2007).
3 By the early 1980s, over 90 percent of English departments responding to a survey stated they used holistic scoring. See Edward M. White, Holistic Scoring: Past Triumphs, Future Challenges 83 in Validating Holistic Scoring for Writing Assessment: Theoretical and Empirical Foundations 83 (Michael M. Williamson & Brian A. Huot eds., Hampton Press, Inc. 1993) [hereinafter Validating Holistic Scoring]. The prevalence of holistic scoring continues. The Scholastic Aptitude Test (SAT), for example, began holistic scoring of essays in 2005.
4 There are many texts that provide an explanation of the holistic evaluation process. See Erika Lindeman, A Rhetoric for Writing Teachers 245-58 (4th ed., Oxford U. Press 2001).
5 We use the term “modern language theory” as shorthand for the description of several linguistic philosophies named by Gerald Wetlaufer: Saussurian linguistics, structuralism, poststructuralism, and semiotics. See Gerald Wetlaufer, Rhetoric and Its Denial in Legal Discourse, 76 Va. L. Rev. 1545, 1546 (1990).
6 For a history of how holistic assessment emerged in twentieth century writing assessment, see Norbert Elliot, Maximino Plata & Paul Zelhart, A Program Development Handbook for the Holistic Assessment of Writing 27-43 (U. Press of Am., Inc. 1990). For a more personal view of the history of holistic assessment, see White, supra note 3.
7 Plato, Apology, in Euthyphro, Apology, Crito, Phaedo 28 (Benjamin Jowett trans., Prometheus Books 1988).
8 In this evaluation, the final judgment may only be a two-point scale (guilty or not guilty; one or zero), but it reflects the myriad of smaller judgments made by the citizen-judges of Athens. It is always important to remember that Socrates, in losing his case, failed to impress the citizenjudges.
9 This opposition is well established in the foundations of forensic rhetoric. Both Aristotle and Cicero also address this classical dilemma: Aristotle dissects the elements of truth, facts, and persuasive art as elements that play a part in forensic judgment. See Aristotle, The Rhetoric and The Poetics of Aristotle 20-23 (W. Rhys Roberts & Ingram Bywater, trans., Random House 1954). And he recommends that orators introduce forensic speech with an appeal to the audience, stating that the content of the speech may seem paradoxical. Id. In creating a fictitious dialogue between a lawyer and a rhetorician, Cicero devotes considerable space to arguing the relationship between legal knowledge, facts, and rhetorical effectiveness. See 3 Marcus Tullius Cicero, 3 De Oratore 169-95 (Loeb Classical Library) (E.W. Sutton & H. Rackham trans., rev. ed., Harv. U. Press 1945).
10 Wetlaufer, supra n. 5, at 1550-52.
11 See e.g. Linda L. Berger, Applying New Rhetoric to Legal Discourse: The Ebb and Flow of Reader and Writer, Text and Context, 49 J. Leg. Educ. 155, 155-56 (1999). Berger suggests that writers can self-impose an internal dialectic in the composition process, alternating positions as writer and reader.
12 For administrative decisions, our experience is that readers need to take two to three times longer for reading and evaluating administrative decisions than analytical essays for other disciplines. In the reader protocols, we state that readers should conduct an “informed” uninterrupted read of the whole decision first. They may put small marks on the decision to keep track of areas they may later wish to return, but the goal is to give readers a chance to form an impressionistic assessment first. But even taking into account the longer time required to read decisions holistically, it is still more time efficient than other assessment methods.
13 At the National Appeals Division, a portion of the annual performance bonus has been based for several years on a holistic assessment of samples submitted by hearing and review officers.
14 The Administrative Procedures Act, 5 U.S.C. § 552(a)(2)(A) (2006), requires federal agencies to make public final opinions made in the adjudication of cases.