III. The Administrative Decision Rubric
In a holistic assessment, we can assume that legal readers will bring their heightened sense of judgment to assessing the quality of legal writing. The legal reading judgment is often reacting to a particular mode of discourse with a prescriptive voice. A reader reacting to the prescriptive voice, often being forced to agree or disagree with the narrator, will find that the assessment of rhetorical effectiveness becomes intertwined with evaluating the truth of the argument or validating the content of the case. The challenge for developing or adapting the rubric for holistic assessment in a legal context, therefore, is to integrate the components of truth and rhetorical effectiveness into that judgment.
In order to be successful in a legal context, holistic assessment must harmonize the impulse to judge legal writing as either a function of truth or content or both or simply on the basis of its rhetorical effectiveness — the dilemma that Socrates first identified. In emphasizing syllogistic logic and other modes of discourse that drive readers to closure, the commitment to these logical elements implies that lawyers are advocating objective truths. Thus, there may be a tendency to judge legal rhetoric based on one kind of language only. The norm for that kind of discourse, therefore, would erroneously become the standard for the rest of the language.15 Thus, a writing assessment tool that neglects to evaluate the truth, which for this discussion, we may now call the apparent truth component of legal writing, would be considered inadequate.
Yet, history and practice show that rhetorical effectiveness often wins the day in legal rhetoric too. The history of law practice confirms that rhetoric is an art that requires skills and techniques that have persuasive impacts on the readers. The various legal subject areas, such as contract, administrative, and criminal law, present different rhetorical challenges that require varying discursive forms. So an assessment instrument must have the flexibility to evaluate the effectiveness of these forms and conventions too.16
It is important, therefore, that a holistic assessment in a legal context, and especially the rubric used, permit prescriptive and persuasive elements to play out on the reader’s battleground for textual meaning and value. They must compete as elements of the discourse with other conventions, such as fact vs. fiction, organization vs. style, or logic vs. passion. If the rubric is constructed carefully, it can integrate elements of legal rhetoric that would normally encompass apparent objective truths or content judgments. Some elements of such a rubric that are candidates for integrating truth and content are case issue, logic, analysis, and organization. These elements in a rubric can include descriptions that legal readers value as conventions when judging the quality of legal writing. These elements are parts of rubrics for other kinds of writing too; their emphasis in reader protocols and pre-assessment training sessions for legal writing can produce the same kind of statistical results as in other writing disciplines.
The proposed holistic rubric for administrative decisions comes from three main sources: 1) shared traits of rubrics from various other analytic essay rubrics in the writing industry; 2) results of surveys about writing problems in the legal field; and 3) our experience with writing and managing the writing of administrative decisions in a large organization.
The rubric is a six-point scale. Mastery scores are four (Competent), five (Strong), and six (Superior). Unsatisfactory scores are three (Marginal), two (Weak), and one (Incompetent). Since at our agency we focus on quality Internet publication of administrative decisions, we often state that we would be “pleased” to see upper-half decisions posted on the web; we would be “embarrassed” to see lower-half decisions posted on the web. These guidelines are simply another way to guide readers to forming an overall impression that resonates with our real world application of holistic assessment. Administrative decisions that fall in the upper half (four, five, six) may have some errors that “distract” readers, but they do not “obscure” for readers the meaning, issue, or basis of a conclusion. Lower-half decisions exhibit characteristics that obscure for the reader the meaning, issue, or basis for a conclusion.
We propose the following five elements be included in an administrative decision evaluation rubric:
Issue: The decision clearly and correctly identifies all matters in dispute upfront and responds specifically to all aspects of the issues throughout the decision.
Organization: The case narrative has a clear beginning, middle, and end, and it connects parts with clear transitions. It has strong ideas to introduce and organize paragraphs.
Analysis & Logic: The decision effectively analyzes all sides of the issues with thoughtfulness and depth. It uses valid logical reasoning that integrates well-chosen facts and regulations to support sound conclusions. It effectively responds to a faithful representation of the parties’ point of view and refutes those views when appropriate.
Style: The decision employs a readable style that is clear and concise for the level of evidence. It demonstrates control of language, including appropriate word choice and sentence variety.
Mechanics: The decision is generally free from errors in mechanics, usage, and sentence structure. It is free from grammar or spelling issues that would be highlighted in Microsoft Word.
For these criteria, the rubric identifies strengths and weaknesses that characterize the gradations of the scores for each element. For example, a decision that scores six tends to fully satisfy the basic definitions of the elements. Using the organization element as an example, a “six” decision has a clear beginning, middle, and end, with clear transitions and strong ideas that introduce paragraphs. A “four” decision, however, may have some unnecessary repetition or “breaks” in the story that may distract, but not confuse, the reader. Or, main ideas in some paragraphs in a “four” decision may not always be evident. In the lower-half range of organization, the decision may be poorly organized or have gaps that confuse the reader. It may also have poor paragraph organization. At the lower end of the organization spectrum, the “story” of the case may lose the reader. There are other gradations for other holistic categories.17
It is important to remember that the criteria guide an overall evaluation of the writing. Holistic assessment does not establish a catalogue of precise individual errors in each category that might appear; instead, the criteria help the reader decide what impact any errors have on the overall quality of the writing sample. With continuous close reading, even a “six” decision can contain minor errors or distractions. Readers learn that upper-half decisions may have errors in one or more elements that distract readers, but still retain their overall coherence and persuasiveness. For lower-half decisions, the writing errors may include such obvious flaws that they obscure the reading experience. The relationship of the elements gives holistic assessment its dynamic life.
Two rubric categories, issue and analysis and logic, address elements of discourse that legal readers particularly value when judging administrative decisions. Both these categories ask legal readers to weigh elements of objectivity and content, aspects of the prescriptive voice that must be integrated into a legal writing holistic rubric. Issues and issue statements are analogous to thesis statements and proposals in other types of analytic writing. Like other rubrics for those essays, the legal writing rubric asks readers to evaluate some rhetorical traits of issue statements. For example, good administrative decisions state issues upfront; the rubric favors specific issue statements over general issue statements; and in exploring both sides of the argument, the rubric calls for the discussion to respond specifically to the issues throughout the decision.18
But for the issue category, the administrative decision rubric also asks the reader to judge whether the writer has “correctly” identified all the matters in dispute. This judgment is often possible, even though the reader has neither access to the case record nor independent knowledge of a case. Decision writers choose to select specific factual patterns, regulatory citations, and organizational patterns for arguments that provide insight into whether the dispute was correctly identified. For example, a writer who states that the issue is about Appellant’s income threshold, but selects facts and argues the rest of the decision around an Appellant’s medical condition, has clearly missed the issue.19 This contradiction will likely confuse the reading experience, thus driving the assessment to an unsatisfactory level. As a component of issue effectiveness, a judgment about issue correctness supports the basic principles of holistic assessment: the elements of the rubric, properly executed, work with each other to impress the reader.
Most legal readers who assess administrative decisions find that the element of issues in the rubric resonates strongly with the element of logic and analysis. The logic and analysis element puts the apparent objective truth at play in writing assessment. While readers, legal or otherwise, want texts to make sense on formal and intuitive levels, legal writing must employ stricter standards of formal logic to achieve this goal. As a result, the administrative decision rubric for legal writing contains language asking a reader to evaluate whether the text demonstrates valid and sound legal syllogisms to support a conclusion or whether facts and analysis of factual patterns aptly apply a rule or law to reach a sound and valid conclusion. These criteria compel readers to determine if writers have adopted appropriate rhetorical modes in stating an objective factual pattern. They also evaluate whether legal citations properly support the premises of legal reasoning displayed in analytical paragraphs.
The rubric refrains from encouraging a specific format or rhetorical pattern for displaying logic or legal reasoning. One such pattern, for example, might be the IRAC method (Issue, Rule, Analysis, Conclusion), or something similar, for displaying logic and demonstrating a valid conclusion by reasoning from a rule through analysis to conclusion on an issue. Formats like IRAC privilege rhetorical form, but it is the logical content of a passage or paragraph that produces a persuasive impression upon the legal reader. Paragraphs written in prescribed forms may very well be unsound, invalid, or confusing.20 Conversely, a text may demonstrate sound and valid logic even if it varies from the form (i.e., all the premises and conclusion are not contained in one paragraph or a sequence of paragraphs). In instances where premises and conclusions may appear to be scattered or fragmented, the distraction to the reader may be traced to the element of style, which encourages writers to use topic sentences with appropriate detail and conclusions, a criterion that supports the rhetorical delivery of logical syllogisms.21
As readers judge the relationship between the elements of issues and logic, we see the connections within elements of the rubric merge with assessing rhetorical effectiveness. Legal reasoning produces conclusions that answer definitively the questions that issues ask. As readers evaluate whether the issues were correct, they see that the writer adopted a mode of logical discourse in an administrative decision that ensured objective reasoning to arrive at conclusions. These elements spill over to more ordinary, but no less important, rhetorical traits too. For example, logical conclusions — the end result of a proper syllogism — become the topic sentences for paragraphs that display deductive reasoning. This analysis supports elements of style and organization in the rubric. The issue questions and the legal syllogisms become the focus of paragraph and decision organization. In the end, the holistic rubric permits both content and rhetorical effectiveness to work together in a reader-writer-text medium to increase perceived quality.
15 See Emily Grant, Toward a Deeper Understanding of Legal Research and Writing as a Developing Profession, 27 Vt. L. Rev. 371, 382-84 (2003). Grant argues that historically the structure of law schools prioritizes speech over writing. In the oral Socratic tradition, writing is one step removed from speech, which is one step closer to thought or objective reality. In order to form valid and sound arguments and conclusions, classical syllogistic logic, for example, depends upon propositions and conclusions that are “true.” And of course it is important to remember that the root of logic is logos, which means speech or truth.
16 See Edward L. Rubin, The Practice and Discourse of Legal Scholarship, 86 Mich. L. Rev. 1835, 1838 (1988). This approach to judgment of discourse is similar to Rubin’s approach to legal scholarship. Rubin argues that “normative discourse,” a system of socially constituted modes of argument shared by a community of readers, may harmonize the opposition between objective truth and rhetorical effectiveness. Id. at 1891-1905. From this viewpoint, both aspects are embedded in the writer’s judgment and the reader’s evaluation of that judgment in the practice of the discourse. Similarly, a suitable holistic evaluation instrument would capture the value of normative discourse.
17 For a full delineation of the NAD-USDA rubric for evaluating administrative decisions, consult the NAD Style Manual, http://www.nad.usda.gov/Forms/NAD%20Style%20Guide%20Manual.pdf (June 2005).
18 At the NAD, we have also used the holistic rubric to evaluate writing for over 150 writing samples submitted by job applicants. Most applicants submit a brief, memorandum, or prior administrative decision. The rubric approach of analysis is quite telling: for example, approximately 20 percent of candidates thus far have submitted samples without explicit issue statements. Of those samples, about half seem to display a “sense of the issue,” either through some other convention or by the reader inferring the issue from the arguments; the other half simply have unfocused legal rhetoric. Yet, virtually all candidates are law school graduates with significant experience practicing in the legal field.
19 We have found that disagreement among the readers about the issue is often the main reason why scoring may be discrepant, especially in instances where one or two readers score the administrative decision as mastery, while a third reader may score it as unsatisfactory. It is important to investigate these discrepancies as part of continuous reliability monitoring.
20 See Kristen K. Robbins, Paradigm Lost: Recapturing Classical Rhetoric to Validate Legal Reasoning, 27 Vt. L. Rev. 483, 483-87 (2003). Robbins makes the excellent point that instruction in the precise mechanics of syllogisms is necessary to understand fully the form of legal reasoning and the basis for short-cut formulaic conventions such as IRAC. At the NAD, we were able to demonstrate that some initial training in classical reasoning and logic contributed to an increase in holistic scores in less than a year’s time.
21 See id. at 487-517. Robbins shows and categorizes some examples of faulty reasoning typical of legal writing. We have gone over these examples, judging them through the lens of the holistic rubric, and have determined that the rubric would take these kinds of reasoning errors into account.