Academic Program Review Orientation

2022-23 Cycle
Program Review Basics
Rationale for Academic Program Review

BOG Regulation 8.015 requires reviews of all academic degree programs

by CIP code and degree program level at least once every seven years.
Reviews must:
• Review mission and purpose of the program within the context of USF’s
mission/strategic plan and the BOG’s strategic plan
• Assess how well the stated student learning outcomes and program objectives
are being met
• Document how assessment results are being used for continuous program
improvement
• Inform strategic planning, program development, and budgeting decisions at the
university and state levels
Components of the Program Review
• Program self-study by CIP and degree program level (e.g., B.A./B.S. Geology, M.S.
Geology, Ph.D. Geology, etc.)
• Department self-study
 Focuses on faculty & resources
• Dean’s report
 Includes questions for external reviewer
• External reviewer’s report
 Includes recommendations
• BOG summary report and improvement plan
 Prepared by department and submitted to ODS-IE for review and submission to BOG
• Progress report on actions taken to implement improvement plan one year after
submission of the BOG summary report
Types of Reviews
On-campus Virtual (Teams) Document Review

• Three-day, in • May be same • Review of

person review format as the on- electronic
meeting faculty, campus review or documents and
staff, students, and some half-day material available
touring facilities, sessions based on online.
etc. external reviewer
availability.
Program Review Timeline for 2022-23
Spring 2022 Summer 2022 Fall 2022 Spring 2023 Summer 2023 Fall 2023

APR Orientation Session Provided by ODS-IE Spring 2022

ODS-IE Sends Data Packets Sent to program August 31, 2022

Due to Jonna DeSantis September 1,
Select External Reviewers 2022

Submit Program & Departmental Self-Studies Due to Dean/Associate Dean November 1, 2022

Dean Submits Report & Self Studies Due to Jonna DeSantis December 1, 2022

Feb. 1 – April 14,

Conduct Site Visit or Online Review with External Reviewer(s) 2023
Due to Jonna 2-3
weeks after visit
Receive External Reviewer Report (e.g., May 5,
2023)

Review External Report (Department, Dean, & Provost) May 19, 2023

Submit Departmental/Faculty Response (Optional) May 31, 2023

Submit BOG Summary Report to ODS-IE Due to Jonna October 1, 2023

Due December
Upload Summary Report to BOG Portal (ODS-IE) 15, 2023

Submit Progress Report about Implementation of Improvement Plan on January 31, 2025 for inclusion in College Review
1-3 Program, Department Self-
Studies, and Dean’s Report
1. Program Self-Study Summary Data Sources from ODS
Students
• Student Enrollments
• Student Term-to-Term Enrollments (5
semesters)
• New Admits by Fall Term
• Enrollments by Gender, Ethnicity,
Nationality, and Student Level
• Degrees awarded by Academic Year,
Gender, Ethnicity, and Student Type
• Student Credit Hours (SCH)
Faculty
• Full-time (tenure / tenure track)
Faculty
• Number of Adjunct Faculty
• Tenure track / non-tenure track
• Faculty by Rank, Ethnicity, and Gender
(schedule meeting with Adam)
• Departmental SCH by Instructor /
Instructor Type
• Graduate and Student Assistantships
 1. Program Self-Study Summary Data Sources from ODS
Assessment Data (System for Assessment Management)
• Annual reports & executive summary data (three-years of data).

GPA & Test Score Data

• Disaggregated by a student identifier (PIDMS) for the number of years available.

Graduating Senior Survey Dashboard

• https://usfweb.usf.edu/ODS/secure/Surveys/GSS/GSS.aspx

Post-graduation Outcomes Hub

• https://usfweb.usf.edu/ODS/Secure/Outcomes/OutcomesHub.aspx

Note: Available behind single sign-on; if access is unavailable, contact Hennadii Balashov (Balashov@usf.edu) for data export.
1. Undergraduate / Graduate Program Self-Study
Section 1 - Academic Program Overview
Overview of degree program’s missions and
purposes.
List major changes since last review and the
rationale for those changes.
Describe modalities by which the program is
offered, including the proportion of the program
available through online learning.
List all instructional sites for face-to-face
instruction.
List competing programs within the Florida SUS.
Describe the structure for student advising for
academic program(s).
Data Source
Undergraduate Academic Catalog: https://catalog.usf.edu/content.php?catoid=13&navoid=1629.
Information provided by the department.
If new, refer to the degree proposal: https://academicplanning.usf.edu/curriculum/tracking.php.
Data provided by the Department / College.
If no online offerings, state that the program is not offered online.
Data provided by the Department / College.
For the Florida SUS, use the link: https://prod.flbog.net:4445/pls/apex/f?p=136:45:0::NO.
For program outside of Florida, draw on experience from professional conferences to identify
competing programs outside the state.
Data provided by the Department / College.
1. Undergraduate / Graduate Program Self-Study
Section 1 - Academic Program Overview Data Source
Provide the program’s criteria for admission (if
applicable).
1. Undergraduate / Graduate Program Self-Study
Section 2 – Industry Demand for the Program Data Source
Summarize current state of the discipline. • Bureau of Labor Statistics
• Occupational Outlook Handbook
• O*Net
• Employ Florida
• Enterprise Florida
• Florida ExpertNet
• Florida Department of Economic Opportunity: Employment Statistics
• Florida Department of Economic Opportunity: Targeted Occupations List

Provide national, state and/or local data that Burning Glass (provided by ODS)
support program need US Department of Labor statistics
FETPIP results from Post-Graduation Outcomes Hub:
https://usfweb.usf.edu/ODS/Secure/Outcomes/OutcomesHub.aspx.

Indicate whether an industry or employer advisory Info provided by the department

council exists…
1. Undergraduate / Graduate Program Self-Study
Section 2 – Student Demand for the Program
Discuss the characteristics of the program’s students
over time.
Reflect on the growth or decline in student
enrollment for your program and comment on the
future potential demand and the program’s ability to
meet this demand.
Describe current recruitment/enrollment strategies
for a diverse body of local, national, and
international students
Data Source
• Number of majors.
• Mix of full and part-time students.
• Mix of FTIC, transfer, and international students.
• Number of students participating in education abroad experiences.
• Number of students participating in undergraduate research experiences.
ODS Data Table: Enrollment.
Information provided by the department.
Information provided by the department.
1. Undergraduate / Graduate Program Self-Study
Section 3 – Teaching and Student Learning Data Source
List the student learning outcomes for the program. System for Assessment Management: http://usfweb.usf.edu/DSS/SAM/.

Summarize the results of the most recent three years System for Assessment Management
of student learning outcome assessments. (Information provided by ODS).

Describe how the results of the assessment of Information provided by the department.
student learning outcomes have been used to
implement program changes.

Describe internal faculty led efforts to enhance: • Innovative pedagogy.

• Student clubs / organizations.
• HIPs.
(Information provided by the department).

Describe the processes used to ensure quality is Information provided by the department.
maintained in the curriculum and in the delivery of
instruction.
1. Undergraduate / Graduate Program Self-Study
Section 3 – Teaching and Student Learning Data Source
3.7 Describe how student learning outcomes Information provided by the department.
assessment ensures that students are professionally
prepared for careers in the field and / or continued
graduate education.
1. Undergraduate / Graduate Program Self-Study

Section 4 – Program Productivity Data Source

Discuss the productivity of the program in terms ODS Data Table: Student Credit Hours & Faculty Data.
of the following:
ODS Reporting & Analytics Hub: https://ods.usf.edu/
• Student credit hours generated.

• Student FTE.

• Degrees awarded (If program was designated

as having “low productivity” in the annual
Program Productivity Review, include the
action plan for improvement).

• Time-to-degree.

• Percent of undergraduates without excess

credit hours.
1. Undergraduate / Graduate Program Self-Study
Section 4 – Program Productivity Data Source
Describe the national reputation or recognition Information provided by the department.
of the program and list the sources of the
reputation or ranking indices.

Provide licensure / certification pass rates for Information provided by the department.
the last three years (if applicable). Include the
number of students attempting and passing the
exam.
Describe any significant student Information provided by the department.
accomplishments.

Provide information on program graduates FETPIP or other related data (https://usfweb.usf.edu/ODS/Secure/Outcomes/OutcomesHub.aspx

including (1) job placement rates, (2) employer ).
satisfaction, and (3) graduate school placement.

Reflect on student perceptions of program The Graduating Senior Survey at https://usfweb.usf.edu/ODS/secure/Surveys/GSS/GSS.aspx.

quality based on appropriate data sources.
1. Undergraduate / Graduate Program Self-Study
Section 5 – Program Strengths and Weaknesses Data Source
List program general / administrative goals and objectives. Information provided by the department.
Reflect on whether these goals had been met during the last
review period.

Discuss potential opportunities in the following areas: Information provided by the department.

• Actions to improve program quality.

• New ventures to increase demand or improve

competitiveness.

• Actions to achieve productivity gains.

• Actions to improve efficiency and reduce cost.

• Other opportunities.
1. Undergraduate / Graduate Program Self-Study
Section 5 – Program Strengths and Weaknesses Data Source
Describe program barriers, threats, and unique Information provided by the department.
vulnerabilities.

Based on the SWOT analysis and current practice, list Information provided by the department.
continuous quality improvement steps the program is taking
to ensure the achievement of stated general / administrative
goals and objectives.
2. Department Self-Studies Report Data Sources
Peer Institutions
• https://usfweb.usf.edu/ODS/UP_Suite/PeerGroups.aspx
Please note that USF’s peers and aspirational peers may not be the department’s peers or aspirational peers .

USF’s National Peer Group USF’s Aspirational Peer Group

• Stony Brook University • Rutgers University
• Temple University • Texas A&M University
• University at Buffalo • University of California Davis
• University of Arizona • University of California Irvine
• University of California Riverside • University of California San Diego
• University of Illinois, Chicago • University of Illinois Urbana-Champaign
• University of Iowa • University of Texas Austin
• University of Kansas • University of Washington
• University of Utah
• Virginia Polytechnic Institute
2. Department Self-Study Report
Section 1 – Department Overview Data Source
Provide a brief history of the department and its Departmental website or department chair’s institutional memory
academic program(s).

Provide the department mission statement and

a description of its alignment with the college
mission, USF Strategic Plan, SUS strategic plan,
and other state priorities.

Describe the departmental vision.

Describe departmental goals for teaching,

research, and service.

Describe the process used to prepare the self-

study including the people involved.
2. Department Self-Study Report
Section 2 - Benchmarking
Compare the department to at least two peer
departments with at least one department at an
AAU peer institution.
Data Source
Select two departments that the department considers to be peers, and select two departments to
which the departments aspires to be a peer. The USF institutional peer comparisons are located at
https://usfweb.usf.edu/ODS/Secure/Performance/KeyPerformanceMetrics.aspx
If using peers not included on the list of USF’s National Public Peers, please provide information to
justify your choice(s).

2.2 Compare the department to at least two

aspirational peer departments…with at least
one department at an AAU peer institution.
2. Department Self-Study Report
Section 3 – Faculty Excellence
Using the data supplied by ODS, provide an
overview of faculty quality and number
including:
• Number of full-time faculty
Data Source
Program and department benchmarks are aspirational and may change on an ongoing basis.
Relevant department or program benchmarks are generally decided in consultation with the
department chair, program director, faculty, and dean’s office. While student data are
appropriate, the focus is usually on faculty-related data (i.e., publications, impact factors, etc.)
All benchmarks should help USF in the pursuit of AAU status.

• Faculty mix (full-time vs. adjuncts)

• Racial / ethnic and gender makeup of the

faculty

• SCH production by faculty type

• Ranks and tenure status

• Areas of expertise
2. Department Self-Study Report
Section 3 – Faculty Excellence Data Source
Describe any efforts to recruit, mentor, and Information provided by Department
retain talented diverse faculty.

Describe how faculty assignments are balanced Information provided by Department

among teaching, research, and services.

Describe the typical teaching load for full-time Information provided by Department
faculty in the program under review. Note any
differences based on tenure status or other
relevant factors.

Give an overview of faculty efforts related to Data extracted from Academic Analytics or other reputable, publicly available sources.
research. • Anticipated future directions
• Community-engaged research
• Research resulting in patents and licensing efforts
2. Department Self-Study Report
Section 3 – Faculty Excellence Data Source
Describe significant professional service Information provided by the Department
activities of the faculty members who contribute
to the program under review: Local,
State, National, and International.
Describe national or international awards or Information provided by the Department
other evidence of noteworthy recognition and /
or other accomplishment received during the
review period by faculty who contribute to the
program under review.
2. Department Self-Study Report
Section 4 – Financial Sustainability Data Source
Provide an overview of the sufficiency of Information provided by the department chair or program director based on known
institutional fiscal, physical, and human departmental practices.
resources to support the program(s) under
review. Include information on:
• Tuition revenue
• Graduate assistantships and waivers (if
applicable)
• Administration support
• Laboratory space
• Laboratory equipment
• Department and classroom facilities
• Office space
• Technology support
• Library resources
• Online course / program support
• Academic Program Advisors
• Staff support
2. Department Self-Study Report
Section 4 – Financial Sustainability Data Source
Describe any non-E&G sources of fund that Information provided by the department chair or program director based on known
support the program(s) under review (e.g. cost departmental practices.
recovery, market rate, INTO, auxiliary funds,
student fees).

Other than university resources, describe any Data provide by ODS.

sources of funds used to support graduate
assistants (e.g. grants, fellowships, donors). For interpretation and other questions, contact Adam Caskie at acaskie@usf.edu.

Discuss the SCH produced by the department to Information provided by the department.
the rest of the university, including service
courses or programs in Enhanced General
Education and interdisciplinary programs.
2. Department Self-Study Report
Section 4 – Financial Sustainability Data Source
List all documented community partnerships Information provided by Department
that are housed in the department.

Discuss any additional unmet resource needs of Information provided by Department

the program(s) under review.
3. Dean’s Report

• The dean’s office will review the program self-

study(ies) and department self-study and provide
commentary and questions for external reviewers.
Note: Specific questions from the dean to evaluators will help guide the evaluator’s
program review.
2. Department Self-Studies Report Data Sources
Academic Analytics
• https://www.usf.edu/provost/faculty/faculty-productivity-tools.aspx

• Academic Analytics provides powerful data and analysis tools that help universities reach their research potential. USF
uses Academic Analytics to allow faculty to find each other, form research networks and compare our programs to our
peer list of institutions. The Academic Analytics database contains scholarly data on over 275,000 faculty members at
over 400 US-based research universities. The data includes research grants, honorific awards, articles, books,
conference proceedings, and more.
Top Oversights in Self-Studies
• Data that should be included in a report are not included.

• Data are included in a report, but they are not summarized in a useful way for the external reviewer.

• Data are summarized appropriately, but are not analyzed or interpreted to create meaning.

• Analysis results are meaningfully interpreted, but the report fails to document how findings were used to
improve the program.

• Detailed descriptions of processes are provided, but with no apparent purpose, result, or
recommendation.

• Response fails to address the question being asked.

4. External Review and Reviewer
Basics and Details
4. External Reviewer’s Report
• ODS-IE provides program and department self-studies and dean’s report (1-
3) to the external reviewer(s) along with guidelines for the external review
report (4).
• The reviewer produces a report that includes recommendations for
improvements that do not focus solely on additional resources. (Program
review is not a budget request process.)
• The report is submitted to Jonna DeSantis in ODS-IE within 2-3 weeks after
the site visit.
• ODS-IE (Jonna) shares report with the department and the dean.
• The department notifies ODS-IE (Jonna) it has accepted the report.
• Upon acceptance of the report, the reviewer is paid for services.
4. External Reviewer’s Report
Selecting external reviewers
• The department identifies potential reviewers based on criteria provided in guidelines.
• The dean reviews potential reviewers.
• ODS-IE, in consultation with the Office of the Provost, confirms eligibility.
• The standard is one reviewer per CIP with three degree programs* and levels. (Requests
for additional reviewers must be made to the Provost by the College Dean.)
*Degree program: an organized curriculum leading to a degree in an area of study with at least one program
major.

Review Format

On-campus Virtual (Teams) Document review

4. External Review Basics

• ODS-IE arranges for payment to reviewers (needs six

weeks notice).
• No reviewers may begin working until an approved
Purchase Order is in the system.
Note: USF Health programs cover all expenses, including travel and stipend.
*Degree program: an organized curriculum leading to a degree in an area of study with at least one program major.
4. External Review Basics
Number of Degree Programs On-campus Virtual (Teams) Document Review

One degree $1,500 plus $1,200 $1,000

program/BOG report $1,000 travel
(e.g., B.A./B.S. Geology) $2,500
Additional degree $500 $400 $375
program/BOG report
(e.g., M.S. Geology)
Additional degree $500 $400 $375
program/BOG report
(e.g., Ph.D. Geology)
Totals $3,500 $2,000 $1,750
4. Typical External Review Timeline
Day 1 Day 2 Day 3
Opening meeting/dinner General meetings and tours Follow-up meetings and exit meeting

• Coordinated and paid for by • Meetings with students, • Meetings coordinated by

ODS-IE. faculty, administrators, the department.
staff, and tour of relevant • Exit meeting coordinated by
academic facilities. ODS-IE.
• Lunch at Top of the Palms –
Up to 5 people including
reviewer. Reservation and
payment coordinated and
paid for by ODS-IE.
• Dinner – Coordinated and
paid for by the academic
department.
4. External Review – Preparing for the visit

• The reviewers are responsible for making their own air and
hotel arrangements.
• The department determines schedule for Day 2 and any
follow-up meetings on Day 3, other than the exit meeting.
• The department provides all ground transportation for the
reviewer, including to and from the airport.
4. External Review – Itinerary Procedure

• Each program review will have an itinerary created in Box

based on the itinerary template.
• It is imperative that any and all updates made to the
itinerary are done in Box and not on a version saved on
local computer.
• Access to the official itinerary will be granted once it is
determined who will be responsible for adding meetings.
• Not required for Document Review
 5-6 BOG Summary Report /
Improvement Plan Progress Report
 Program Review Timeline for 2022-23
Spring 2022 Summer 2022 Fall 2022 Spring 2023 Summer 2023 Fall 2023

APR Orientation Session Provided by ODS-IE Spring 2022

Due to Jonna DeSantis September 1,

Select External Reviewers 2022

Submit Program & Departmental Self-Studies Due to Dean/Associate Dean November 1, 2022

Dean Submits Report & Self Studies Due to Jonna DeSantis December 1, 2022

Feb. 1 – April 14,

Conduct Site Visit or Online Review with External Reviewer(s) 2023
Due to Jonna 2-3
weeks after visit
Receive External Reviewer Report (e.g., May 5,
2023)

Review External Report (Department, Dean, & Provost) May 19, 2023

Submit Departmental/Faculty Response (Optional) May 31, 2023

Submit BOG Summary Report to ODS-IE Due to Jonna October 1, 2023

Due December
Upload Summary Report to BOG Portal (ODS-IE) 15, 2023

Submit Progress Report about Implementation of Improvement Plan on January 31, 2025 for inclusion in College Review
5. BOG Summary Report (DUE Oct. 1, 2023)
1. BACHELOR’S DEGREES ONLY: An Academic Learning Compact is available for each reviewed baccalaureate program.
(Current ALC’s available at http://usfweb.usf.edu/DSS/SAM/Public/ALCs.aspx)

2. Was the program review conducted in conjunction with any specialized or programmatic accreditation reviews? (If
yes, provide a description of the reviewer)

3. Date of the last Academic Program Review for this program.

4. Briefly describe major changes made since the previous program review.

5. Summarize current strengths of the program.

6. Summarize current weaknesses of the program.

7. Summarize recommendations made by the external evaluator.

8. Describe improvement plans made as a result of the review.

Note: https://www.usf.edu/ods/documents/academic-programs/master-bog-summary-report-2020-12-10.docx
6. Improvement Plans Progress Report (DUE Jan. 31,
2025)

• Status of improvement plan as described in summary report

• Improvement plan covers the implementation of program
action items and recommendations
• Improvement Plan is due to 1-year + month from
submission of Summary Report (Due to BOG Dec 2023)

Note: https://www.usf.edu/ods/documents/academic-programs/master-bog-summary-report-2020-12-10.docx
Academic Program Review Support Staff
• Jonna DeSantis, Executive Administrative Specialist, coordinates recruiting evaluators, document management, and
scheduling; jonnad@usf.edu

• Laura Brunkow, Fiscal and Business Specialist, coordinates contracting with and payment to evaluators; laurab@usf.edu

• Hennadii Balashov, Statistical Data Analyst, provides data tables and ODS-related data; balashov@usf.edu

• Joseph Boyd, Assistant Director, coordinates APR Process, serves as St. Petersburg campus liaison, jboyd6@usf.edu

• Rebecca Gibbons, Assistant Director, serves as Sarasota-Manatee campus liaison, revaclav@usf.edu

• Christopher Combie, Director, ODS-IE, facilitates academic program review process & troubleshoots problems;
combie@usf.edu
 Questions?

Electronic files of everything discussed are posted

on the Academic Program Review Website:
http://www.usf.edu/ods/iep/institutional-effective
ness/academic-programs/academic-program-revie
w.aspx
.

Thank you for your time and attention.