The Power of Data for Decision Making andthe Appropriate Use of Analytics in HigherEducation Settings
Michele J. Hansen, Ph.D., Assistant Vice Chancellor Institutional Research and DecisionSupport, IUPUI
The Power of Data for Decision Making andthe Appropriate Use of Analytics in HigherEducation Settings
Michele J. Hansen, Ph.D., Assistant Vice Chancellor Institutional Research and DecisionSupport, IUPUI
• Role of data and analytics in higher education today and in the future
• Definitions and sources of data
• Use of data to improve student learning and success
• Ethics, social justice, security, and privacy
• Questions and discussion
• 
Enhance student achievement
• Plan courses and curriculum
• Recruit and retain students
• Optimizetheschedulingof classrooms
• Understandlevels ofstudent engagement
• The ability to make effective decisions is crucial if an institution of higher education is going to continuously improve student learning and success.
• Data helps decision makers evaluate alternatives, make resource allocations, and make informed choices.
• Must have reliable and timely data (quantitative andqualitative) upon which to make decisions.
• The development of effective data management and analysis
techniques is of central importance.
• Many decision makers find that using data is no easytask as they find themselves inundated with nearly overwhelming amounts of data.
• Improve data processing speed.
• Allow us to summarizemultiple data points.
• Visualization platforms – Tableau.
• Improve access via self-service tools.
Decision Support
• More complex the task, more difficult to
replace with technology enabled tools.
• Decide on research design andeven appropriate statistical testor algorithm.
• Understand complex data questions posed by decision makers.
• Consider ethical use of datafor decision making.
Statement of Aspirational Practice ForInstitutional Research – Association ofInstitutional Research (AIR)
• “Data are everywhere across institutions of higher education, and access to analytical tools and reporting software means that a wide array of higher education employees can be actively involved in converting data into decision-support information.”
• “The demand for data to inform decisions in postsecondary education is greater than ever before. Colleges and universities have significantly increased capacity to collect and store data about student and institutional performance, yet few institutions have adequate capacity for converting data into information needed by decision makers.”
Student Focus
Leadership for IR Function
IUPUI Selected as 1 of 10 Founding Institutions
Statement of Aspirational Practice
Structures and Leadership for IR
Expanded Definition of Decision Makers
Leverage Technology-DataVisualization Platforms such asTableau for Accessible Self-Service Data Tools
InstitutionalResearch and Decision Support
Contains highly interactive dashboards allow users to drill down and filter to allow detailed exploration of key indicators associated with the IUPUI Strategic Plan.
Ø 
Engage in efforts to understand the anatomy of decision making across campus (who makes decisions, when, how, and what data is needed).
Ø Building data literacy, evaluation, and assessment capacity across IUPUI so that information exploration, interpretation, and analysis are used to support evidence-based decision making and improve institutional effectiveness.
Ø Provide consultation on assessment of student learning, program evaluation, survey research methods, and advanced statistical analysis procedures.
Ø Deliver training and data tools that allow decision
makers to leverage data and information.
11
• Institutional research is a broad category of research activitiesconducted at schools, colleges and universities to inform campus decision-making and planning in areas such as admissions, financial aid, curriculum, enrollment management, student success and learning, staffing, student life, finance, facilities, athletics, andalumni relations.
• Typically involves research conducted for internaldecision making, planning, and external accountability reporting.
• Purpose is primarily to improve institutional effectiveness and not to generalizable research or inquiry.
 |
National Data SourcesAssessment
• Assessment is often defined as a continuous cycle of improvement and is comprised of a number of features: establishing clear, measurable expected outcomes of student learning; ensuring that students have sufficient opportunities to achieve those outcomes; systematically gathering, analyzing, and interpreting evidence to determine how well student learning matches expectations; and using the resulting information to understand and improve student learning” (Suskie, 2009, p. 4).
• Statistical methods involved in carrying out a study include planning, designing, collecting data, analyzing, drawing meaningful interpretation and reporting of theresearch findings.
• Gives meaning and often involves inferences.
• Requires and an understanding of quantitative and qualitative variables, measures of central tendency, sample size estimation, power analysis and statistical errors/assumptions.
• Requires a proper design of the study (understanding of research
methods) and choice of a suitable statistical test.
• Improper statistical methods may result in erroneous conclusions which may lead to unethical practice.
• Poor Performance in first semester or Earning DWFI in a course
• Low high school or transfer in GPA (lower than 3.00)
• Under-Resourced (high levels of unmet financial aid, low-income)
• Late Registration Date
• Not having Academic Honors Diploma or Rigorous High School
Curriculum
• Attending part-time and not enrolling in 15 or more credit hours
• Not Placing into Credit Bearing Math
• Transferring in with few hours with no degree
• First Generation College Student
• Not Participating in High Impact Practices and Early Interventions
First Year (FYS, Themed Learning Communities, Summer Bridge)
• Living Off-Campus
• Living Alone or With Others Not Attending IUPUI
• External commitments (working for pay off-campus, commuting, taking care of dependents and household responsibilities)
• Low self-efficacy, sense of belonging, commitment toIUPUI (intent to transfer)
2017 Themed Learning CommunityImpact on First Year GPA:ANCOVA Results
|
|
N |
Avg. FallGPA |
Adjusted FallGPA* |
|
TLC |
936 |
2.76 |
2.79 |
|
Non-Participants |
2374 |
2.74 |
2.73 |
|
Overall |
3310 |
2.74 |
|
Note 1: Only Full-Time FYS participants. Students who withdrew from a TLC were counted as non-participants.
Excluding students who were missing data on one or more covariates.
Note 2. Differences were statistically significant based on Analysis of Covariance(ANCOVA) results (p <. 048). Note 3: Partial Eta Squared indicated a very a smalleffect size.
* Covariates included in the model were High School GPA, SAT Score, Enrollment Date(proxy for student motivation and commitment), and Income Level (received a Pell Grant orNot dummy coded where 1 = Received Pell Grant and 0
= Did Not Receive a Pell Grant) and Gender.
Diversity Enrichment andAchievement Program (DEAP)RetentionFall-Fall Retention – Retained IUPUI IN
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
82%
80%
African American Latinx Two or MoreRaces
DEAP (Total N=178) 
Nonparticipants (TotalN=730)
• 18 DEAP Students Received Housing Stipends in 2016 Retention Rate was 89%(Fall-to-Fall)
• 31 DEAP Students Received Housing Stipends in 2017 Retention Rate was 94%(Fall-to-Spring)
• 
DEAP students also participate in living-learning communities and SummerBridge.
2014 2015 2016
Retained IUPUI IN Not Retained IUPUI IN
Retained and Not Retained Ns (2014=2162;1033), (2015=2236; 995),(2016=2365; 1038)
Recent IUPUI StudentExperience Survey Spring 2018• 
Onaverage,29% ofstudentsAgreedorStronglyAgreedwithstatementsthatindicatetheyperceivethat otherpeople atIUPUIhave agrowthmindset.
• Faculty andstaff do playa role inbolsteringadaptivemindsetsaboutintelligence—which canpowerfullyshape students’ own growth mindset and, in turn,theiracademicoutcomes
• Examples of items:
• “Ingeneral,mostpeopleatIUPUIbelievethatsomestudentsaresmart, while others are not”
• “In general, most people at IUPUI seem to believe that students have a certain amount ofintelligence, and they really can’t do much to change it.”
N=769 Undergraduates (16%)
• Faculty and staff do play a role in bolstering adaptive mindsetsabout intelligence— which can powerfully shape students’ own growth mindset and, in turn, their academic outcomes
• Provide support for learning
• Set high standards and convey that we are motivated to help studentsattain them (journey taking together)
• Give sense of purpose (applying learning experience to life and realworld problems)
• Foster growth and not fixed.
• Set clear expectations and giving constructive, clear feedback onlearning
• Communicate that IUPUI is a place that student belongs (notjust a place for other students)
IUPUI
• Understand data and limitations.
• Not relying on single data point for telling whole story.
• Know data definitions.
• Understand of how data collected,sources, and research methods.
• Correlations does not mean causation.
• Data analytics is the science of drawing insights from raw data sources. Many ofthe methods and techniques used in data analytics are automated into mechanicalprocesses and algorithms that organize raw data for human decision making.
• Data analytics used to understand patterns of data that may otherwise be lost in the mass of information.
• Term used to describe statistical and mathematical data analysis that clusters, segments, scores and predicts the scenarios that are most likely to happen.
• “A set of [business intelligence]technologies that uncovers relationships and patterns within large volumes of data that can be used to predict behavior and events”.
• “Predictive analytics is forward-looking,
using past events to anticipate the future.”
van Barneveld, A., Arnold, K.E., & Campbell, J.P. (2012).
• Combine technology, research, and consulting to help
you identify your most at-risk populations, coordinate cross- campus resources so students canfind the help they need,
and analyze the data that informs your decision making.
• Provider of research, enterprise technology and data-enabled services for education institutions.
• Student Success Collaborative membership of more than 500 colleges and universities across the country working together to improve student outcomes and thestudent experience.
I Overview Success Progress History Class Info Major Explorer More T
|
Course Grade Repeated Withdrawn Missed cumulative
D/F Courses Courses successMarkers GPA
7 ... 9 "f'" 8 ... 5 ... 2.38 ...
••n••-••••-•••n••••••••-•••n•••••••••••••n•••
Total Credits Earned
Credit Completion %
at this Institution
Predicted Risk Level
Moderate
Bachelor of Science
Schoolof Science
ALTERNATEID
Other Majo_(s) CLASSIFICATIO
Libera] Arts Undergraduate Sociology BA
Bachelor of Arts
School o Liberal Arts
Senior
MOSTRECENTENROLLME T
Falll 2018
• Billions of data points being generated every minute of every day by humans, computers and technological devices – creating a real-time digital footprint of our lives with every credit card swipe, phone use, Google search, Facebook post, and more.
• With availability of this ocean of data, how can we use it to better understand and our world andserve our needs.
• While college and universities doing cutting edge research on Big Data and educating data scientists,not using it as much as other industries to innovateour academic institutions.
Lane & Finsel, 2014
“From the dawn of civilization to 2003, humans created five exabytes worth of data. As of 2013 humans produced this same amount of information every two minutes” (Miller &Chapin, 2013 as cited by Lane and Finsel, 2014)
• Typically field of Data Science used for processing Big Data - Data Science field used to tackle big data. an umbrella term that encompasses data analytics, data mining, machine learning, and several other related disciplines.
• Involves gathering data from multiple sources and applies machine learning, predictive analytics, and sentiment analysis to extract critical information from the collected data sets.
• Machine learning used as practice of using algorithms to learn from data and then forecast future trends for that topic. Traditional machine learning software comprised ofstatistical analysis and predictive analysis that are used to spot patterns and catch hidden insights based on perceived data (used by Facebook).
• Machine learning is a method of data analysis thatautomates analytical model building. It is a branchof artificial intelligence based on the idea thatsystems can learn from data, identify patterns and make decisions with minimal human intervention. (SAS)
• An algorithm is a set of heuristics and calculations that creates a model from data. To create a model, the algorithm first analyzes the data you provide, looking for specific types of patterns or trends. The algorithm uses the results of this analysis over many iterations to find the optimal parameters for creating the mining model. These parameters are then applied across the entire data set to extract actionable patterns and detailedstatistics.
• “The interpretation of a wide range of data produced by and gathered on behalf of students in order to assess academic progress, predict future performance, and spot potential issues.”
• “The use of predictive modeling and other advanced analytic techniques to help target instructional, curricular, and support resources to support the achievement of specific learning goals”
• Allows instructors to tailor educational opportunities to each
student’s level of need and ability.
• Can be used assess curricula, programs, and institution
• Source: van Barneveld, A., Arnold, K.E., & Campbell, J.P. (2012).
• Grades on assignments and exams – progress
• Engagement (logging on and page views)
• Attendance
• Activity
• Chat rooms
• University-and course-level learningoutcomes - scores on rubrics
• E-texts also provide powerful markup and
interaction tools.
• Highlighting, shared notes, questions, and answers.
• Research found that higher engagement with e-texts (reading and highlighting) correlated with higher course grades (Abaci, Quick, andMorrone, 2017) .
Sample Projects
Using Analytics to Evaluate Influences on Student Learning Outcomes in a GenEd Science Course (G131, Oceans & OurGlobal Environment): Phase II of an Analytics ApproachSimon Brassell (Geological Sciences)
Seeks to continue to utilize analytics on student demographics and grade records in combination with data on their performance in the GenEd NMSclass G131 “Oceans and Our Global Environment” to assess howperformance in the class and its range of assignments may be related tospecific student characteristics.
Determinants of Student Attrition Michael Kaganovich (Economics)
The proposed research will focus on the factors behind IU students’decisions to make substantial changes in their studies at IU: to discontinue enrollment altogether (i.e., to drop out) or to switch from one major to another.
HJiflt fl/I 5' UTULL
WEAPONS OF
MATH DESTRUCTION
IIIIW BIEi llil,TA INtREil,EiEi!I JIIEUUALllY
Alila T H:EATrlHi aEMIIC'lll!.C'I'
CATHY O'NElt
• Predictive models and algorithms are increasingly the tools used to make decisions that affect people’s lives ---where they go to school, whether they get a loan, how much they pay for health insurance, what type of sentence people receive when convicted of a crime.
• Decisions being made by mathematical models rather than
humans.
• Ideally, the mathematical models are unbiasedand lead to greater fairness. Not True!
• Many models used today are mysterious, unregulated, and uncontested with no feedback or correction mechanisms. Can be wrong.
• Algorithms if left unchecked essentially increase inequality
creating “toxic cocktail for democracy.”
• Used for harm rather than good.
• Algorithms reinforce discrimination and widen
inequality,
• Use people’s fear and trust of mathematics to prevent
them from asking questions.
• Rely on proxies (proxies are easier to manipulate than complicated reality they represent).
• “Algorithms that are important, secret and destructive”.
• Affect large numbers of people, are entirelyopaque, and destroy lives.
• Models are opinions embedded in mathematics.
• Baseball
• Amazon
• Predictive modeling used to help provide
resources for students
• Drown the bunnies!
• US News and World Report Rankings (algorithms based on proxies and rankings become destiny).
• Marketing by for profit colleges(ads that pinpoint people in great need and sell them false and overpriced promises – predatory ads).
• Admissions decisions.
• Transparency (not relying on black boxes).
• Continuously update.
• Assumptions and conclusions clear.
• Rely on actual data rather than proxies.
• People being modeled understand the process and
understand the models objective.
• Use to help rather than harm.
• Family Educational Rights and Privacy Act (FERPA)
• Institutional Review Boards (IRB) – generalizable research
• Stories of data violations
• Data Governance
• IUPUI Student Data Advisory Council and Faculty/Staff Data Advisory Council
• Ownership of data such as course evaluations and LMS/Canvas data
• Explaining use of data
• Data definitions and metadata
• Clear notes and sources
• Understanding limitations
• Explore prior to analyses and use
• Clear explanations of methods used to analyze or organize data
• Wealth of data available for decision
making
• Value of data-based decision making
• Data literacy
• Ethical Use
• Theory-based methods
Final Thought• In order for students to be productive citizens in a world in which lower skilled labor is being replaced by computers and robots, we need an educational shift focused and need to rebalance our curriculum to develop students with “creative mindsets and the mental elasticity to invent, discover, or create something valuable to society rather than concerned solely with “topping up students' minds with high-octane facts.”
• New skills: data literacy to manage the flow of big data, and technological literacy to know how their machines work, but human literacy, from the humanities, communication, and design, to function as a human being in a world populated with artificial intelligence advanced technologies.
Aoun (2018), author of Robot- Proof Higher Education in the Age of Artificial Intelligence,
MicheleJ.Hansen,Ph.D.AssistantViceChancellormjhansen@iupui.edu317-278-2618
Institutional Research and Decision Support
irds.iupui.edu
Contact us with questions or requests for information!
IUPUI
• Aoun, J. E. .(2017). Robot-Proof: Higher Education in the Age of
Artificial Intelligence. MIT Press.
• Abaci, S., Quick, J., & Morrone, A. S. Studentengagement with e- textbooks: What the data tell us. Educause Review 52 (4)
• J. E. Lane (Ed.), Building a smarter university: Big data, innovation,
and analytics. Albany, NY: SUNY Press. Foss, L. H. (2014).
• O’Neil, C. (2016). Weapons of Math Destruction:How Big Data Increases Inequality and ThreatensDemocracy. New York, NY: Crown Publishers
• Swing, R. L.& Ross, L. E. (2016). A new vision for institutional
research, Change: The Magazine of Higher Learning, 48 (2), 6-13.
• van Barneveld, A., Arnold, K.E., & Campbell, J.P. (2012). Analytics in higher education: establishing a common language [White paper]. Boulder, CO: EDUCAUSE Learning Initiative. Available at https://library.educause.edu/resources/2012/1/analytics-in-higher- education-establishing-a-common-language
• Designing College for Everyone. Brief written by the College Transition Collaborative.
• Leveraging Mindset Science to Design Educational Environments that Nurture People’s Natural Drive to
• Designing Supportive Learning Environments. Video created by the Mindset Scholars Network.
• The New Science of Wise Psychological Interventions. Journal article by Gregory Walton, published in Current Directions in Psychological Science.
• Social-Psychological Interventions in Education: They’re Not Magic. Journal article by David Yeager and Gregory Walton, published in Review of Educational Research.
• Broadening Participation in the Life Sciences with Social-Psychological Interventions. Journal article by Yoi Tibbetts, Judith Harackiewicz, Stacy Priniski, and Elizabeth Canning, published in CBE Life Sciences Education.