About This Episode
In episode 18 of Data and AI Heads, host Ian Allison speaks with Cynthia Pekron about a rigorous controlled study she conducted with researchers from Harvard Business School and Vanderbilt to measure how machine learning predictions, including Shapley value explanations, affected the decisions of expert knowledge workers. The experiment divided analysts into groups with varying levels of ML information access and tracked outcomes at three months and one year, revealing that while overall decision quality held steady across groups, analyst confidence increased almost linearly with greater ML transparency. Critically, less tenured analysts who leaned heavily on ML outputs showed signs of eroding independent reasoning, raising important questions about how organizations develop talent in an AI-augmented environment.
Cynthia introduces the concept of over-automation, drawing on her own career experience where hiding certain data attributes from human oversight led to a quiet but significant quality degradation that took nine to twelve months to surface. She offers a practical framework for AI project selection: distinguish between undifferentiated processes where heavy automation is safe and IP-rich, expertise-dependent workflows where institutional knowledge is the competitive moat. The conversation closes with actionable guidance on establishing pre-deployment benchmarks, partnering with BI teams on core KPIs, and designing governance structures that protect both quality and the long-term growth of expert talent.
Key Takeaways
- Access to Shapley values, which explain the attribution of data inputs to ML predictions, increased analyst confidence almost linearly even when analysts did not change their final decisions based on the ML output.
- Less tenured knowledge workers who relied heavily on ML predictions produced rationale that reviewing committees found less compelling, signaling a risk of critical thinking erosion when junior experts over-rely on AI tools.
- Over-automation can create a silent quality degradation: in one real-world example, removing human visibility into certain data attributes caused a measurable drop in quality that did not appear until nine to twelve months later.
- Organizations should apply a build-versus-buy mindset to automation decisions, automating undifferentiated processes freely while protecting expertise-dependent workflows that constitute the firm's competitive moat.
- Before deploying any AI initiative, teams must establish a clear benchmark for current quality and efficiency, including historical outcome data over sufficient time horizons, to enable meaningful measurement of AI impact.
- Change management for AI is bidirectional: senior experts need reassurance that AI amplifies rather than replaces their science, while junior staff need structured workflows that still require them to develop independent reasoning.
- Partnering closely with the BI team to define core KPIs before deployment, and building those KPIs into the new system from the start, is essential to producing discernible, defensible ROI results.
- Automating the least interesting parts of expert roles and creating a clear, demonstrated path toward higher-value work is one of the most effective strategies for earning practitioner trust and managing AI change resistance.
What We Cover
Frequently Asked Questions
What did the controlled study on AI and expert knowledge workers actually find?
The study divided analysts into groups with different levels of access to ML predictions and Shapley value explanations. Overall decision quality was statistically similar across all groups at three-month and one-year follow-ups. However, analyst confidence increased almost linearly with greater ML transparency, and less tenured analysts who leaned heavily on ML outputs produced rationale that review committees found less compelling, indicating a risk of critical thinking erosion.
What are Shapley values and why do they matter for knowledge worker confidence?
Shapley values show which data attributes contributed to an ML prediction and by how much, providing an attribution explanation for the model's output. In the study, analysts who had access to Shapley values felt more confident in their decisions even when they did not directly use the ML prediction as their conclusion. Understanding why the model reached a prediction appeared to help experts articulate and validate their own reasoning.
What is over-automation and how does it damage organizational quality?
Over-automation occurs when human oversight is removed from processes where understanding the underlying logic still matters for quality outcomes. Cynthia Pekron described a case early in her career where certain data attributes were hidden from human review as part of an automated system. Over time, the team lost understanding of why those attributes were important, and quality degraded, but the problem did not surface for nine to twelve months, making it difficult to trace back to the automation decision.
How should organizations decide which processes are safe to automate heavily versus which require protecting human expertise?
The guiding principle is whether the process is part of the firm's competitive moat or IP. Undifferentiated processes, such as moving data between tables or standard ETL work, are safe candidates for heavy automation because customers do not choose a firm based on excellence in those tasks. Expertise-dependent workflows where human judgment, institutional knowledge, and research quality are the actual product should be automated more carefully to avoid commoditizing the firm's core differentiator.
What steps should a data or AI leader take before deploying AI to ensure measurable ROI?
Leaders should first define what value looks like for the specific use case, whether that is efficiency, quality, or both, and then establish a clear benchmark of current performance using historical data over a sufficient time horizon. They should partner with the BI team to set core KPIs and build those metrics into the new system from the start. For quality-sensitive or regulated environments, running the initiative in an experimental or sandboxed governance structure before production deployment reduces risk and builds a credible evidence base.
How can organizations manage change resistance when introducing AI to expert practitioners?
Two strategies proved effective in Cynthia Pekron's experience. First, focusing initial automation on the least interesting or most tedious parts of expert roles and demonstrating a clear path toward more meaningful, higher-value work addresses fears about job displacement. Second, for highly credentialed experts who are deeply devoted to their field, framing AI as a tool that magnifies and curates their expertise rather than one that replaces their science or art helps earn trust and shift attitudes over time.
What is the risk of rushing AI deployment without accounting for institutional knowledge loss?
Organizations that move quickly on AI initiatives to stay competitive may inadvertently create a knowledge gap that degrades performance over a one-to-three-year horizon. If junior employees rely heavily on AI outputs before developing independent critical thinking, the firm loses the depth of expertise that actually differentiates its products or services. This means a competitor who invests more deliberately in talent development alongside AI could gain a long-term quality advantage, turning the rushed deployment into a strategic liability.
About the Guest
Cynthia Pekron
Cynthia Pekron is a practitioner with deep experience building human-in-the-loop machine learning platforms for expert knowledge workers. She has collaborated with academic researchers at Harvard Business School and Vanderbilt on controlled studies measuring how AI and machine learning predictions affect the decisions, confidence, and long-term development of expert analysts. Her work spans applied ML, knowledge management, change management for AI adoption, and the design of governance frameworks that protect institutional knowledge while enabling measurable efficiency and quality gains. She brings a practitioner perspective on the tension between automation and expertise preservation, informed by hands-on experience with both the risks of over-automation and the organizational dynamics of introducing AI to highly credentialed subject matter experts.
Connect on LinkedIn →More Episodes