Section 4.5 Studying and working with LLMs

Compare two ways of studying

When evaluating a study assistant, a satisfaction rating and a mastery test answer different questions. Prepare exercises paired by topic and difficulty. Vary the order of conditions when several students take part. Give each condition comparable time and access to the same course resources.

Keep two measurements separate

The first rubric evaluates the model’s answer for validity, suitability of the hint and compliance with the request. The second measures what a person can produce independently on an unseen question. A model can deliver high-scoring answers that lower the demands of the activity. A brief diagnostic question can instead help a student identify an error and continue their own work.

Two study conditions lead to the same kind of unaided assessment. The quality of model replies is measured within each condition.
Figure 4.5. The common test compares the conditions. Model responses are evaluated separately.

Interpret a small experiment

Save initial attempts and errors, then inspect concepts where results differ. A student working alone can alternate matched sets and retake an assessment after a delay. Session order, previous practice and actual difficulty remain possible influences. The results characterize this experiment. A conclusion about a population needs a suitable design and sample.

Suggested deliverable. In one page, report the question, both conditions, exercises, criteria and a representative error. Summarize what changed in the student’s independent work.