4 Read Like A Robo-Grader: Developing Audience Awareness
Alise Lamoreaux
Before beginning to think about what words would influence a Robo-Grader, think about how people are persuaded. For example, how do kids get their parents to do what they want? Or, how do families decide what products to buy? Consider the following situations and what type of evidence might be important for each of the cases below:
Situation #1
Your school is looking at some of its policies to make changes. You have been asked to provide evidence about whether homework is harmful or helpful.
- How would you present this to your school? Consider what type of information you would want to provide. What sort of information would make good evidence in this case?
- Whose point of view should the presentation use? Students or teachers?
- How would you present the same information to students as you would to teachers? Would you change anything you said based on your audience?
- Is there a difference in effectiveness of evidence based on the audience? Why?
Situation #2
The City Council for where you live is considering a ban on “vaping”. You have been asked to provide evidence on this topic.
- How would you present this information to the City Council?
- What type of information and evidence do you think would be important to include for this type of a decision? What would make good evidence for this decision?
- How would you present this information to your friends to let them know what was happening with the City Council?
- Would there be a difference in your presentation based on the audience you were trying to influence? Why?
Situation #3
Your school is working on revising its Student Code of Conduct. Plagiarism has been a problem that students don’t seem to understand. The committee you are serving on is dealing with the question, “Is copying someone else’s work ever acceptable?”
- What type of information and evidence do you think would be important to include for this type of decision? What would make good evidence for this decision?
- How would you present this information to the students of the school?
- How would you present this information to the local community?
- Would there be a difference in your presentation based on the audience you were trying to influence? Why?
Evidence can be categorized in many ways. Think about the above situations and the type of evidence best suited to each of those situations. Here are examples of 6 major types/categories.
- Pathos: Pathos involves emotional appeals. Language that shows emotions or feeling conveys the pathos. The goal of the evidence is to sway the emotions of the decision maker. An example of pathos might be, “Don’t be the last person on the block to have their lawn treated – you don’t want to be the laughing-stock of your community!” Or, “You’ll make the right decision because you have something that not many people do: you have heart.”
- Ethos: Ethos tries to show that the person providing the evidence is believable. Expert witnesses in a trial are an example of ethos-the insinuation is that a psychiatrist’s opinion about a person’s state of mind should carry more weight with a jury, or that a forensic scientist should be able to interpret evidence better than the jury.
- Logos: Logos involves studies, data, charts, and logic to back up the statements being put forth. An example of Logos would be, “More than one hundred peer-reviewed studies have been conducted over the past decade, and none of them suggests that this is an effective treatment for hair loss.”
- Kairos: Kairos involves an argument that creates a state of urgency. An example is this quote from Sir Thomas Moore, “This is the right time, and this is the right thing.“
- Big Names: Big names involves using names of experts or well-known people who support your position. Think of celebrity product endorsements or causes they support.
- Testimony: Testimony can be a personal story as support for why something should happen. It can be referred to as anecdotal evidence. Personal proof as a way of supporting the claim.
How will the automated essay grader recognize evidence?
The AEG will be looking for key words that signal the presence of evidence, such as words that link ideas together and show progression of thoughts, or words that show relationships. That is why the word “because” has now been called a “power” word. It links 2 ideas together and implies a relationship of dependency.
Examples of words that link argumentative or persuasive stances together could be:
Not only that, but …
Not only are they …, they are also …
They are not …, nor are they …
There are various/several/many reasons for this.
First, … / Firstly, …
Second, … / Secondly, …
Moreover, … / Furthermore, … / In addition, …
Another significant point is that …
Finally, …
On the one hand, … On the other hand, …
In contrast to this is …
Because of …
That is why …
After all, …
The reason is that …
In that respect …
The result of this is that …
Another aspect/point is that …
It is because …
Although it is true that … it would be wrong to claim that …
That may sometimes be true, but …
One could argue that …, but …
Examples of words that show additional information is being added or a conclusion being drawn would be:
Most probably …
It appears to be …
It is important to mention that…
As indicated …
In other words, …
So, all in all it is believed that…
(In) summing up it can be said that …
In conclusion
Therefore,
In short,
To conclude,
The evidence highlights, or has shown
The strength of … is that
These examples are not meant to be complete lists of words demonstrating connection of ideas, but rather ideas to build from. AEG will be able to recognize synonyms for these words as well. It’s important to understand the patterns of words the AEG will be looking for.
Signposting Sentences
Signposting sentences can be thought of as explanations to the logical organization of the argument. Signposts help guide a reader, human or Robo, through the response. Signposts are helpful elements of each paragraph. They can be thought of as linking words or short phrases. Signposts are a good way to quantify what the response will do. Signposts are like symbols on a road map that make it easier for a reader to know at what stage the response is currently, and where it is going next. Examples of signpost include, but are not limited to, the following lists of words:
Highlighting or emphasizing a point
Importantly, … Indeed, … In fact, … More importantly, … Furthermore, … Moreover, … It is also important to highlight …
Changing direction or creating a comparison
However, … Rather, … In contrast, … Conversely, … On one hand, … On the other hand, … In comparison, … Compared to … Another point to consider is …
Adding a similar point
Similarly, … Likewise, … Again, … Also, …
Summarizing
Finally, … Lastly, … In conclusion, … To summarize, … In summary, … Overall, … The three main points are
Being more specific
In particular, … In relation to … More specifically, … With respect to … In terms of …
Giving an example
For instance, … For example, … this can be illustrated by … …, namely, … …, such as …
Acknowledging something and moving to a different point
Although … Even though … Despite … Notwithstanding …
Following a line of reasoning
Therefore, … Subsequently, … Hence … Consequently, … Accordingly, … As a result, … As a consequence, To this end,
Stop Words
Another important list of words to be aware of when an automated essay grader is involved are stop words. These are commonly used words that search engines are programmed to ignore when indexing word entries. Stop words are deemed irrelevant for searching purposes because they occur frequently in the language. To save both time and space, stop words are ignored. On the website GitHub Gist, a site that is commonly used to house open source projects (https://gist.github.com/sebleier/554280), a list of the stop words for Natural Language Tool Kit (NLTK) can be found. The initial list provided there includes 127 words, many of which are pronouns and conjunctions. Automated essay graders can be trained to read for word pairings, which may be important to realize since a word like “because” appears on the NLTK word list, indicating this “power” word may not actually be read due to the frequency of its occurrence in the English language. A more unique synonym could be a better word choice when the Robo-grader is the audience. Once again, due to proprietary information, there is no way to be sure of the stop words that are being filtered for by the automated essay reader; however, becoming aware of the existence of stop words can help students think carefully about the vocabulary they select. For instance, the words “I” and “think” are both on the stop word list for NLKT, which indicates that those words would not even be indexed and thus not “read”.
One tip to help people increase awareness of their word selection and usage is to use a word processing program, like Microsoft Word, to assess the reading level associated with the piece of writing the person has created. In addition, students can assess the number of sentences used in the writing, the average number of words used per sentence, the average number of characters per word used, for example, to critically examine the details of their writing.
Another aspect of sentence organization to consider is where the key terminology is presented in the writing. AEG will be looking for matches, like the game of Concentration, and the sooner it finds matches to the data it is looking for, the sooner value can be associated with the writing being evaluated. Sentence frames may assist people in configuring their writing in a manner that will be positively seen by the automated essay grader in a fast and efficient manner.
Sentence Frames
Sentence frames and starter sentences are a common suggestion for use in crafting the format of argumentative essays. The frameworks are designed to help students move through the organizational language to create fluency for the reader. It’s important to remember that AEG is looking for patterns it sees in the test essays from which it has been trained. Without really knowing anything about the test set of essays, definitive statements are difficult to make. Intellectual property rights keep the test set of essays elusive; however, an inference could be made that it is likely that some of the test set essays include sentence frames. If that is the case, the AEG reader will rate those sentences as a match to the test set of essays and points will be generated based on the match that occurs.
One question that comes to mind is, what level of test set essays used sentence frames? Top-scoring essays? Essays that receive minimum passing scores? AEG likes unique words, so one strategy could be to examine the sentence frame suggestions and develop alternative ways to state the same framework. The following sentences are examples of using sentence frames. This list is not intended to be a complete list of sentence frames, just a place to start from.
Sentence Frames to Introduce the topic:
- The general argument made by __________ in his/her work ______________ is that _______________ because __________________________.
- Although _____________________ (believes, demonstrates, argues) that ______________, _________________ supports/provides the clearest evidence _______________________.
- A key factor in both _________________________ can be attributed to _________________________________.
- When comparing the two positions in this article, __________________ provides the clearest evidence that ___________________________________.
- Looking at the arguments regarding _________________, it is clear that ___________.
Sentence frames to introduce issues:
- The issue of _______________is a complex one. What it is about is__________.
- The question everyone’s asking is ,______? Here’s the controversy: _______________.
- We all need to consider________________. The debate is about ___________________.
- The issue to grapple with is____________. The problem is ________________________.
- We need to determine if _______________because___________________.
- It will be important to decide ___________because__________________.
Sentence frames to demonstrate the counter argument (Opposing Side’s Position):
- People who disagree may claim that … ”state the opposing side’s position”…
- Critics may claim that …” state the opposing side’s position” … .
- Some people may argue that ___________________.
- A possible concern they may raise is that __________________.
Demonstrate Why The Opposing Argument is Strong
- This opinion could be possible due to __________________.
- They may have a strong argument as a result of ______________.
Rebuttal (Explain Why Their Argument is Weak)
- This argument is wrong since _______________.
- The evidence, however, overwhelmingly supports the argument that _____________________.
- On the contrary _________
It becomes clear that word selection, vocabulary depth, and essay organization are key components of success when the audience is automated essay graders. Understanding who the target audience of the writing will be is always crucial to receiving positive judgement from the reader; however, while AEG can mirror the results of human graders, it is not proven that they arrive at the similar conclusions for the same reasons. It’s important to remember that some of the fundamental principles of “good” writing will be ignored by automated essay graders. Even though the “stop words” can be ignored by AEG, they are essential to human communication in standard English. Additionally, they may impact the total word count of the piece of writing, which is another aspect of the overall feature rating system.
