Finetune

Generate®
 The Ultimate Item Generation Tool
Generate is a hybrid AI-Human tool designed to supercharge
Subject Matter Experts (SMEs).

Rather than starting with a blank page or an overused template,

Generate produces brand new item ideas (including stimuli,
questions, options), targeted specifically for your assessment,
increasing productivity and creativity many times over.

Copyright 2023. All Rights Reserved

 Combining AI and Human Knowledge

Security

Speed

Quality

Copyright 2023. All Rights Reserved

 Generate Addresses The Needs of The Assessment
Industry While Keeping Content Completely Secure

You have your own AI model, trained by your best SMEs Enable SMEs to focus on their expertise, while increasing
productivity and creativity

Speeds up item authoring production (highest captured so

far = 30x), while simultaneously increasing item quality, Develop item stems, options, key, distractors or full
diversity and richness of item pools passages from scratch in seconds

Your more experienced SMEs will flourish with this new Improve test security and reduce cheating concerns with
productivity and creativity enhancement. Their usage will bigger and more frequently updated banks of novel
help train the model to continually improve and create materials
stronger content

Train the AI platform for increased item development

Provides consistency and guidance to less experienced or accuracy, customizable for different cognitive complexity
trained SMEs with built in best item writing practices, fairness levels
improvements, all aligned with your required style

Kickstart a formative, personalized, adaptive learning

Highly targetable for efficient addressing of item pool gaps assessment or test prep program

Copyright 2023. All Rights Reserved

 Generate is NOT ChatGPT

ChatGPT is an OpenAI
product that utilizes Large
Language Models (LLMs) in
an automated chatbot
format. Generate is
superior to ChatGPT in the
following ways:

Copyright 2023. All Rights Reserved

 Design Principles We Live By
Design for Workflow,
NOT for AI
> AI assisted regeneration of
stems provides options if you
don’t like the initial output
> AI will regenerate directly in
alignment with changes
Copyright 2023. All Rights Reserved
Center the Human in
the Loop
> Finetune’s mission empowers
content creators and instructors
> Identify patterns of feedback
early
> Subject Matter Experts (SMEs) are
treated as equal partners in model
development, alongside data
scientists and stakeholders
> It’s all about ‘Agency’
Build Trustworthiness
through Transparency
> LLMs have ’Trust Issues’
> User feedback helps answers
questions – ‘How do I know where
this came from?’
> Important features of Finetune
Generate: AI-Assisted Reference
Lookup
> Important features of Finetune
Catalog: Rationale and Confidence
Scores
> Ensure that each test owner has
unique model(s) specific to their
content with strict privacy and
security features
 What Else Can Generate Do?
Work with virtually all item types that have text or images,
compose brand new passages, scenarios, and case studies,
along with alt text ideas for graphics
Work with math, including scientific symbols and equations
Has best item writing practices built in, with a creativity bar
available to increase variability of items, including AI-
assisted reference features available
SMEs train the model and benefit from AI assisted
regeneration of stems to provide options if you don’t like
what is given and will then regenerate directly in alignment
with changes you’ve made
All text elements are editable (stimulus, stem, options) with
customizable metrics for readability and word count, with
reorder options and rationales
Admin features allow customized access to content and
permissions for folders

AI-assisted search can determine which area is lacking in

content and target which topics to write questions about Multiple item download formats (e.g., CSV, QTI, text) and
item preview in WYSIWYG or you can preview complete item
sets
Can import stimulus material including copyrighted material,
alt text, and transcripts
Supports a clone feature so you can make variations of an
item
Can be formatted specifically to your “book of knowledge”,
targeting area, technical or grade level of content, while also
considering competency, cognitive complexity levels (e.g,
Bloom level) and learning objectives
Copyright 2023. All Rights Reserved
 We Do More Than Connect to
Generative Models
We emphasize psychometric validity of generated assessment content

Our AI scientists and psychometricians work together within a principled design

framework to isolate constructs of interest and measure them reliably

Our applications enhance the assessment workflow, not just provide a wrapper for
generative AI

Finetune is a company you can trust. We protect your intellectual property and develop
explainable AI models you can understand and interact with

Copyright 2023. All Rights Reserved

 Plus, You Own Your
Intellectual Property
Finetune does not assert ownership or
copyright on the content you create using
our engine. You will have exclusive rights to
use AI models we build for your content,
and these will not be made available to
other parties. The underlying AI models and
technology is owned by Finetune

Copyright 2023. All Rights Reserved

 Finetune Generate Process
Phase Description What you provide (*note we can work
with you without these artifacts)

Model Development We learn about your specific needs for your content ● Test blueprint
development process, item types, levels of cognitive ● Style guide
complexity, level of specific needed. We assess your ● Exemplar items
needs and determine the scope and how many ● Business goals
models you need and then we build them. ● Knowledge on how you run your
business now
● If relevant, reference or book of
knowledge materials

Implementation After creating the models, we will ensure that your ● Item writers
team is set up for success. We will train and check in ● Time to train the item writers
with them to make sure their questions are answered.
Prove out efficacy in a data driven way.

Ongoing License We will embark on an agreement and provide If desired, check-ins to help your team
continued support to your teams. make the best use of the platform.

Copyright 2023. All Rights Reserved

 Model Development Phase
Goal
Process
What you provide (*note we can
work with you without these
artifacts)
Output
Copyright 2023. All Rights Reserved
The model development process is where our AI scientists and measurement scientists assess your needs and
create a domain specific and impactful model specifically tailored to your needs.
1. Your team will transfer the available data to our team.
2. Our measurement team assesses the materials you provide and tailors a plan to generate psychometrically
valid assessment content. They account for the domains you need covered, the cognitive complexity of the
items, the lexile levels, and other qualities assessment professionals care deeply about.
3. The AI scientists take the measurement team’s assessment and work with a collection of large language
models, but also proprietary techniques that have taken years to develop.
4. The measurement team then takes the output of the AI team’s work and provides feedback that meets
your outlined specifications.
● Test blueprint
● Style guide
● Exemplar items
● Business goals
● Knowledge on how you run your business now
● If relevant, reference or book of knowledge materials
Model(s) and submodels available on Finetune Generate interface
 Implementation
Goal To set you up for success during your implementation

Process 1. Our team will meet with the team who will be using Generate to author items through the interface. About an hour is needed to
teach the item authors how to generate items, make edits, metatag items, save items, and how to download items in the desired
format.
2. After users have some time with Generate, we will want to conduct a quick check -in to assure things are working well or if
adjustments need to be made (best practices are being used, the AI model is delivering as expected, clear up any issues).
3. At this point, the item authors now will have full access to the Generate Interface where your team members produce
representative passages, items, content that they can edit and finalize iteratively on the spot. You own all items developed.
4. If desired, if there is initial feedback that we need to address, the AI scientists will update the model.

What you provide ● Item writers

● Time to train your item writers

Output Your item writers/SMEs will be trained and ready to utilize Finetune Generate

Copyright 2023. All Rights Reserved

 Ongoing License
Ongoing license means that your team has access to your models and the support you
need to make the most of them.

Goal You will have access to your models and the support you need to make the most of them

Process 1. If desired, check-ins to make sure that your team is using the platform correctly
2. Updates on new features
3. Continued support

What you ● Item writers

provide

Output You will supercharge your SMEs to speed up authoring up to 10x

Copyright 2023. All Rights Reserved

 What is a Model?
Our process and pricing hinges on the concept of a “model”. We want to walk you through what we
mean.

Technically speaking, a Generate “model" is a collection of specific sets of instructions and

generation parameters, all of which target a specific assessment.

What that means practically is that a model covers a broad domain most often equal to a test
blueprint for an assessment. You will have access to many submodels, which appears as options
for item writers within the Generate interface. When choosing that submodel, the item writer can
expect generated items of their desired form (e.g. multiple choice questions) related to their
specified domain within a set of requirements (e.g. no banned text).

Copyright 2023. All Rights Reserved

 Examples with sample blueprint
K12, medical, certification examples

Copyright 2023. All Rights Reserved

 Example Test Blueprint
1 Model/High Complex
> Multiple Reporting Categories
> Crossing several subdomains
> Associated with multiple task models
and cognitive complexities

Copyright 2023. All Rights Reserved

 Example of PN
Version of Exam
> Equally complex
> Questions must be different from RN
model because of the different
responsibilities between RN and PN

Copyright 2023. All Rights Reserved

 Process Dev OPs Cyber Security
Novice

Intermediate

Expert

Copyright 2023. All Rights Reserved

 ELA REQUIREMENTS:

Reading:
-Key Ideas and Details Models for Grade Bands
-Craft and Structure
-Integrity, knowledge, and ideas Grades 3-5 1 Model
Grades 6-8 1 Model
Writing: Grades 9-12 1 Model
-Test Types
-Production of Writing
-Research

Depth Of Knowledge = Webb's 1-3

Map to Achievement Level Descriptors

Copyright 2023. All Rights Reserved

 Mathematics Test Requirements:

Operations/Algebraic thinking
Models for Grade Bands
Number 2 Operators
X (Mathematical Grades 3-5 1 Model
Measure/Data/Stat Practices) Grades 6-8 1 Model
Grades 9-12 1 Model
Geometry

Depth Of Knowledge = Webb's 1-3

Map to Achievement Level Descriptors

Copyright 2023. All Rights Reserved

 Science (K-12)

Performance Expectation

Models for Grade Bands

Science and Cross
English
Disciplinary
Cutting Grades 3-5 2 Models
Core Ideas
Practices Concepts Grades 6-8 2 Models
Grades 9-12 4 Models

DOK = Webbs 1-3

Map to Achievement Level Descriptors

Copyright 2023. All Rights Reserved

 Productivity Gains
Generate typically can help you realize productivity a range of productivity gains of
2.5x – 30x. Yes, that is a wide range! The more complex, authentic and sophisticated
the item, the greater the time savings.

Here are some of the variables:

> Cognitive complexity of items

> Item Type - does it require a passage or opening stimuli?
> Would you like references and/or rationales?

Copyright 2023. All Rights Reserved

 Efficacy Study Traditional vs Generate
Quality
1. Productivity:
> SME manual: 45 min/item
> Using Generate: 17.6 min/item
> Productivity Increase: ~156% (2.5x faster)

2. Quality Ratings (94 items):

> Control Items (47): 5.61
> Generate Items (47) : 5.79
> Note: Ratings range 1 - 6 (rating of 4 is an
acceptable item)

Generate Bottomline:
> Faster (2.5x)
> Exceeded Quality (slightly higher)
> Increased creativity and uniqueness

Copyright 2023. All Rights Reserved

 AI Solves Critical Problems Across The Entire Content and
Assessment Ecosystem
Difficulty in adapting quickly to “Industrial Cheating Industry” Demand for greater emphasis on
providing new material and ever- accelerated by limited question banks personalized modular assessment
requires MORE assessment
changing content. and assessment content.
content.

Assessment and Learning Content needs

Learner-centric assessments and
to be properly tagged to align with ever-
Adaptive Learning both require
changing sets of competencies and
superb conceptual tagging and
taxonomies. The incumbent process is
massive quantities of targeted items.
labor intensive, difficult and slow.
Copyright 2023. All Rights Reserved 24
 How to Determine
Short-Term Success
Here are the potential metrics that can be collected during the
pilot process. Once the pilot metrics are identified, targets for
those metrics can be identified for the pilot.

SME Engagement Metrics Item Writing Metrics (Items Ready for Formal Item Review)
● Number of items saved
● Number of SMEs trained
● Time spent per item saved
● Amount of time spent in the Generate software
● Number of passages/scenarios created
● Feedback provided within the software
● Time spent per passage/scenario saved
● Feedback provided within the SME discussion with
Finetune
Quality Metrics
SME Satisfaction Metrics
● SME Survey
● SME Survey Data
● Item Review Results
● Pretest Results

Copyright 2023. All Rights Reserved

 How to Determine
Long-Term Success

SME Impact Item Quality Impact

● # of SMEs ● Pretest Statistics
● SME Availability ● Sensitivity and Bias
● SME Retention ● Differential Item Functioning (DIF)
Item Bank Impact Security
● Breadth and Depth of Item Banks ● Item Exposure
● Item Overages ● Expanded Market
● Item Pretesting

Copyright 2023. All Rights Reserved

27 27
 We look forward to working with you

Steve Shapiro Simmy Ziv-el Sara Vispoel Charles Foster Jesse Hamer Brad Bolender Wendy Gavin
CEO CBDO CALO Lead AI Scientist Lead AI Scientist Principal Product Manager
Measurement
Scientist
Contact Us:

Simmy@Finetunelearning.com

Sshapiro@Finetunelearning.com

28
 NOV

2014 2017 2020 2021 2022 2022 2023

Embarked on Members of the Early Access to GPT-2 Invented Acquisition by Prometric! Release of ChatGPT Generate and
partnership with the Finetune team begin then GPT-3. Finetune Catalog™. and World Learns Catalog are well
College Board AP doing research on Twenty deep Pilots with about LLMs. established in the
Division to build out LLM’s. Invented Large Orgs in Ed Publishing, market and continue
AP Formative Finetune Generate® High Stakes Assessments in to evolve.
Assessment Platform (Patent Pending). Licensure, Credentialing and
that serves over 3M Education and Test Prep.
students and 200K
instructors.

29
 Thank you

Copyright 2023. All Rights Reserved

31 31