Module 7: Evaluation for NLP systems

Module 7: Evaluation for NLP systems#

Theme#

Evaluation for NLP systems

Essential Question#

Why are output quality and factuality hard to measure?

Module Components#

  • Book prose: conceptual framing, domain scenario, methods, and failure modes

  • Assignment: evidence-backed production of a specific artifact

  • Slides: presentation sequence for seminar or lecture delivery

  • Narration: spoken version of the slide flow

  • Rubric: criteria for evaluating the module artifact

  • Notebook: executable lab aligned with the module theme using synthetic support messages, retrieval snippets, intent labels, and factuality checks

Module Artifact#

NLP evaluation packet with task framing, retrieval/evaluation design, and deployment guardrails focused on evaluation for nlp systems: Create an evaluation set with rubrics and automated checks.

Professional Setting#

Students work as if advising a product team evaluating an NLP workflow before using it in customer-facing communication. Their work must be intelligible to product manager, support lead, privacy reviewer, and model evaluator.

Use This Module in Order#

  1. Read the learning chapter.

  2. Review the slide deck with the matching narration.

  3. In Populi, open the private student-repository link for this course and enter modules/module-7.

  4. Clone the repository once or open its Codespace/Colab copy; run lab.ipynb and complete exercise.ipynb there.

  5. Self-check with the rubric, commit and push the work, then submit exactly what Populi requests.