Overview
Large question banks power randomized online quizzes, common final exams across sections, placement tests and professional certificates. Built quickly, they tend to contain items that test trivia, give away the answer through grammatical clues, or have more than one defensible key. Once in use, those items quietly reduce the fairness and reliability of every test drawn from the bank.
We start with a test blueprint (a table of specifications) that states which outcomes and content areas the bank covers, in what proportions, and at what cognitive levels. Items are then written to agreed guidelines — a single clear problem in the stem, plausible distractors, no 'all of the above' — and pass through technical, content and sensitivity review.
After the bank has been used, we analyze item performance with classical test theory: difficulty, discrimination, distractor functioning and overall reliability. These statistics show which items to keep, revise or retire. For high-stakes tests with larger samples, we can discuss whether item response theory modeling is appropriate.
Who it’s for and the problems it solves
This is for you if you are…
- Departments running common exams across many sections or campuses
- Online programs that need randomized question pools for quizzes
- Institutions developing admission, placement or exit exams
- Professional bodies and training centers building certification exams
- Ministries and agencies developing national or sector assessments
Common problems we help with
- The bank is large but most items test recall of isolated facts
- Students share questions because pools are too small
- Some items have ambiguous keys and generate appeals every term
- Nobody knows which items are too easy, too hard or misleading
- Items written by different staff vary wildly in style and quality
What’s included — and what isn’t
Included
- Test blueprint linking content areas and learning outcomes to item counts and cognitive levels
- Item-writing guidelines tailored to your subject and item types (single best answer, multiple response, matching, numerical, short answer, scenario-based sets)
- Item writer training workshop and feedback on writers' first items
- Item drafting with your subject experts, or review and editing of items they write
- Multi-stage review: technical flaws, content accuracy (by your experts), language clarity for multilingual test-takers, and sensitivity/bias
- Metadata tagging: outcome, topic, cognitive level, difficulty estimate, author, review status
- Import into Moodle, Canvas or Blackboard question banks, or export in formats such as QTI or GIFT where supported
- Item analysis after administration: difficulty (p-value), discrimination (point-biserial or upper–lower index), distractor analysis and reliability (KR-20 or Cronbach's alpha)
- Bank maintenance plan: revision cycles, retirement rules and security practices
Not included
- Administering or proctoring exams
- Taking exams on anyone's behalf or supplying answer keys to students
- Setting pass marks for high-stakes certification without a standard-setting process owned by your institution
- Claiming content accuracy for items your subject experts have not reviewed
What you receive
- Test blueprint
- A table of specifications with content and outcome weights, cognitive-level distribution and item counts per form or pool.
- Item-writing guide
- A concise guide with rules, good and flawed examples from your subject, and a review checklist.
- Reviewed item bank
- Items with keys, rationales, references to course materials and metadata, in a spreadsheet master file plus LMS-importable files.
- Item analysis report
- Statistics for each item, flags for revision or retirement, and suggested edits in plain language.
- Maintenance and security plan
- Rules for exposure control, versioning and who can edit the master bank.
How it works
Blueprint
Agree scope, purpose and stakes of the test, then build the blueprint with the course or exam committee.
Writer preparation
Workshop for item writers and a calibration round in which everyone writes and critiques a small set of items.
Writing and review
Items are drafted in batches and passed through technical, content and sensitivity review, with a tracked status for each item.
Load and pilot
Items are imported to the LMS or test platform; where possible, new items are piloted unscored alongside operational items.
Analyze and refine
After administration we run item analysis, recommend revisions and update the bank and its metadata.
Item-writing and psychometric methods
- Table of specifications
- Content-by-cognitive-level grid ensuring the bank samples the domain as intended, not just what is easy to write.
- Item-writing guidelines
- Widely used multiple-choice rules: clear stem, one best answer, homogeneous and plausible distractors, no negatives without emphasis, no cueing.
- Classical item statistics
- Difficulty (proportion correct), discrimination (point-biserial correlation or upper–lower group difference) and distractor selection rates by ability group.
- Reliability
- KR-20 for dichotomous items or Cronbach's alpha, interpreted in light of test length and purpose.
- Item response theory (where appropriate)
- Rasch or two-parameter models for large, high-stakes banks that need equating across forms; we advise on data requirements before recommending it.
- Tools
- LMS item analysis reports, Excel and R or Python scripts for custom statistics.
Writing items that test thinking, not memory
Scenario-based items — a short case, data table or clinical vignette followed by a question that requires applying a concept — can assess application and analysis while still being machine-scored. We help writers move from 'Which of the following is the definition of X?' to 'Given this situation, what should happen next and why?'
Reading item statistics sensibly
Statistics flag items for human review; they do not make decisions. A very difficult item may be excellent if it covers an advanced outcome and discriminates well. An item with negative discrimination — stronger students getting it wrong more often — usually has a flawed key or an ambiguous stem and needs immediate review.
- Low discrimination plus extreme difficulty: check for miskeying or content not taught
- A distractor chosen by nobody: replace it with a plausible misconception
- A distractor attracting high scorers: check whether it is also correct
- Small samples: treat statistics as provisional and look at trends across administrations
What the work can look like
Illustrative example — fictional data
Illustrative excerpt: item analysis flags
A fictional 60-item common final exam in introductory accounting taken by 480 students across five sections. Statistics are invented.
Summary
KR-20 reliability 0.84. Mean difficulty 0.63. Seven items flagged: three with discrimination below 0.10, one negative, three with non-functioning distractors.
Item Q27 (negative discrimination)
p = 0.41, point-biserial −0.08. Top-quartile students mostly chose option C. Review found option C is also correct under the revenue recognition rule taught in week 9. Recommendation: rescore Q27 and rewrite the stem to specify the context.
Item Q12 (non-functioning distractor)
Option D chosen by 1% of students. Suggested replacement based on a common error seen in homework: confusing gross and net figures.
Illustrative example with invented statistics; interpretation always depends on your test's purpose and sample.
What you’ll need to provide
- Course or exam outcomes and syllabus, plus any existing items
- Subject matter experts for writing or content review
- Target LMS or test platform and import requirements
- Anonymized response data (item-level scores) for item analysis
- Policies on exam security, accommodations and language of assessment
You don’t need everything ready to ask for a quote — tell us what you have and we’ll say what’s missing.
What affects price and turnaround
Price depends on
- Number of items and item types (scenario sets and numerical items take longer than simple items)
- Whether we draft items or review and edit items your staff write
- Number of review stages and languages
- LMS import and the need for custom formats
- Depth of item analysis and number of administrations analyzed
Turnaround depends on
- Subject expert availability for content review
- Exam date and any required pilot administration before operational use
Every project is quoted individually. How we price our work.
Questions people ask before starting
How many items should a bank contain?
It depends on how tests are assembled. For randomized online quizzes, each pool usually needs several times the number of items drawn so students see different combinations. The blueprint sets targets per content area rather than a single total.
Can you write items in our specialist field?
We can draft items from your materials, but a qualified subject expert on your side must review content accuracy and approve every item. For many institutions the most efficient model is that faculty write and we train, edit and review.
How is bank security handled?
Master files are shared only with named staff through channels you choose, and we recommend version control, restricted editing rights and exposure limits. We do not keep copies after the engagement unless the contract says otherwise.
Do we need item response theory?
Not usually. Classical item analysis is sufficient for most course exams. IRT becomes worthwhile for large, high-stakes programs that need equated forms and have enough test-takers per item.
Can you help with standard setting for pass marks?
We can explain methods such as Angoff or bookmark and facilitate a panel, but the pass mark is an institutional decision made by the panel and approved through your governance.