World language departments around the globe are continually faced with the task of advising and placing freshmen and transfer students into the appropriate courses. This decision is often based primarily upon the number of classes the students have previously taken in the language. However, placement on this basis does not take into account several important factors which determine a student’s actual ability in the language. These factors include the effectiveness of the student’s past teacher(s), the curriculum covered, and extracurricular or other out-of-class exposure to the language.
The increasingly popular Seal of Biliteracy and its associated tests do offer an alternative solution, but traditionally, universities and colleges have relied on an interview-style test to assess language ability. While there are several commercial test products that use interview-style tasks, these tests can be expensive and can take a while to be scored. As a result, many organizations have moved language testing processes in-house and use their existing language resources—if they have any—to conduct language screening interviews.
Throughout my career, I have worked with language teachers, administrators, and other professionals to develop their skills in this type of assessment. If you plan to use an in-house solution for language assessment like an interview-style test, here are some points to consider.
Focus Questions on Language Function
I love getting to know people, and an interview assessment is a great way to do that. However, it is important to remain focused on the purpose of this particular interview, which is to elicit evidence of language ability. Questions should be specific to that objective. Here are a list of key functions that you should include in a well-rounded language ability interview:
- A flex plan approach
Have a plan, but plan for flexibility, too. I will usually create a grid with several questions that target a particular function so that I can adapt during the interview as I see what tasks or content areas are the best fit for the person being interviewed. - Pre-instructions
They say if you start right, it is harder to go wrong. Before the actual interview begins, it is critical to give explicit instructions about the goal of the interview. Something along the lines of:
“Of course, I’m looking forward to getting to know you! However, I am mostly interested in getting to know your abilities as an English (Spanish/Chinese, etc.) language speaker. With that in mind, as you answer questions, answer them fully and with sufficient detail that it shows what you can do in English. If you think that a question will not give me good information about your abilities, maybe because it is not something you know a lot about or have experience with, let me know. Just say, ‘Could we talk about something else?’ I hope it is okay with you but it is essential that I record our conversation. That will allow me to review it afterward. While we are talking, I will be focusing more on my part. When I listen afterward, I will be evaluating your performance. Any questions, or should we jump right in?” - Warm up
It is good to ease the person being interviewed—and, frankly, yourself as the interviewer—into the task. A warm up should be easy. It is usually safe to start at an intermediate level. One of the most helpful rules for knowing you are at the intermediate level is at the heart of the word intermediate—ME. Intermediate speakers can talk about themselves. Try this:
“Let’s let you get comfortable talking. Tell me about yourself. What are your interests beyond using English? Tell me about your language-learning journey.” - Past tense
If you are measuring English, past tense narration is a good indicator of language ability. Here is an example:
“Tell me about a time when you moved to a new place. Include detail from before, during, and right after the move. I am interested in some of the specific things you did to set up life in this new place.”
When you listen to their response, you will be listening to confirm that the order of the events is clear, well-marked, and accurate in its form. - Description
An important language function that the student will need to manage is giving an appropriate amount of detail. There are three magic words that I use several times in an interview to get at this function of description: “Tell me more.” This phrase encourages the student to draw out additional descriptive details. Some people are talkers; some people are not. “Tell me more” is a natural way to coax out a person’s language ability so that you are measuring it and not their personality. - Redirection
It can also be helpful—at least one time in the interview—to interrupt a description with a phrase like, “Actually, I was hoping to hear more about _”. If you asked them to talk about a famous landmark in their hometown, you may interrupt their description of its physical characteristics and say, “Actually I was hoping to hear more about its history.” Not only does this elicit more breadth from their description but seeing how they handle the shift and interruption can be very insightful. - Process
Identifying whether an individual can communicate a familiar process to someone who is unfamiliar with it is another critical language application to assess for. One way to do this is by asking them to first identify a place that they go to once a week or so. Then, ask them to give you directions for how you would get to that place from their home or a common landmark like a subway station or an airport. Tell them that you will take notes and then, after they finish, recall what details you noted. Instruct them to correct or clarify anything that you missed. - Numbers
Effectively communicating strings of numbers and letters is a strong indicator of language ability that can be helpful to elicit in an interview. There are many possible strings that you could have them produce, but it is helpful to include an unfamiliar or unpracticed string. Most wrappers or products like books, or even dollar bills, have a unique string (often near the barcode) on them.
You can provide them with the object if you are in person. When over the phone, you can ask them to grab a book or wrapper or dollar bill and read out loud the string of letters or numbers that identify it. Write the string down as they communicate it. After they have read it, read it back to them, but change one of the characters. Identifying whether they can confirm and correct the errored string effectively is helpful.
Record the Interviews and Rate the Performance after the Fact
Even when questions are preplanned, conducting an interview in a way that is consistent and fair requires a lot of concentration on your behalf as an interviewer. It is difficult—if not impossible—to split concentration between executing the interview and simultaneously making precise and reliable evaluations about the language performance of a student. Taking a break between the stress of giving the interview and playing it back for scoring will ensure your placement is more reliable. It will also provide you the opportunity to more objectively review your own execution of the interview and improve your skill going forward.
Know Your Limits and Pace Yourself
Although we have identified a few things that will make your efforts more sustainable, a single interviewer can only do so many interviews before their fatigue will start to impact the placement of the student being interviewed. In order to avoid this, take breaks and spread the interviews out over an appropriate amount of time. An interviewer who does not do this will start to suffer from intra-rater reliability issues. This means an interview that they do or a placement they assign early in their effort will be notably different from an interview completed or placement assigned later in the cycle.
Calibrate Multiple Interviewers
Most situations will require multiple interviewers, as you are typically placing hundreds of students each semester. While this increases the capacity for how many students can be assessed at one time, it also increases risks to reliability. In addition to extensive training before interviews begin, ongoing calibration is necessary. Having a preplanned structure to the interview will help consistency, but before anyone scores an interview, they should listen to one or two previously scored samples and assign them a score. If the score they assign does not match the previously determined score, they should practice with a few more recordings to ensure that they are consistent with established standards.
If This Sounds Like a Lot of Work, Time, and Money, It Is…
An interview-style approach has a reasonably low start-up cost, but it is unrelenting in its costs to maintain—particularly when reliability matters. Not only are there direct expenses in hiring, training, and maintaining the right administrators to execute these interviews, but there are also indirect expenses in using a testing method that is simply less reliable than other forms of testing.
A test is reliable when, under different circumstances with different test-takers, it would return consistent results. Like a digital weight scale that produces the same outcome each time it measures a one-kilogram weight, a language test must produce the same score each time a student measures ability. An interview approach to language testing, even at its most refined, simply introduces too much potential variance for its reliability not to be at risk.
Placing a student in the incorrect class doesn’t just cost them time and tuition; administrators and teachers lose time and money, too. The time spent in the wrong class and the lengthy process of interviewing and placing again add up quickly, resulting in productivity loss for all involved.
While these tips for interview-style testing are certainly useful, by using this less-reliable assessment method, you are conceding that you will make wrong decisions more frequently. Even well-trained and well-supported interviewers and raters introduce error in both components. Fortunately, an interview-based approach is not the only solution to the problem of testing students for language ability. AI-powered language testing solutions have improved greatly in their accessibility and accuracy.
AI-powered language testing solutions use technology to elicit language performance. This presents significant advantages in efficiency and scalability. An interview approach can really only assess students one at a time per interviewer. Most computer-based testing will be able to test all students who need to be tested at the same time and—very importantly—test them all under very similar conditions.
Such tests achieve high levels of accuracy by reducing threats to reliability in both the elicitation and evaluation components of their assessment. Through the use of innovative methods to gather and analyze data about a student’s speaking ability, they are able to predict ability with improved levels of accuracy over interview-style assessments—but in 15 minutes or less, much more reliably, and much more inexpensively.
Some of these tests can be used completely remotely. If we have learned anything from 2020, it is that a solution is only as good as it is flexible.
Coordinating an interview approach to language screening adds further complexity at a time when such complexity is increasingly costly. Artificial intelligence helps to simplify and improve the process so no more time and money are wasted wondering which students are properly placed.
Judson Hart has spent 16 years working in English language assessment, teaching, and administration. As director of curriculum at Emmersion, he advances company strategy through developing and furthering innovations in language assessment. Before joining Emmersion, he was the coordinator of assessment and technology at Brigham Young University’s English Language Center.
