Training for Technology: Adoption and Productive Use of Generative AI in Legal Analysis

arXiv:2603.04982v2 Announce Type: replace-cross
Abstract: Can targeted user training unlock the productive potential of generative artificial intelligence in professional settings? We study this question using a randomized experiment in which 164 law students completed an issue-spotting examination under one of three conditions: no GenAI access, optional access to a large language model (LLM), or LLM access with a brief training intervention.
Untrained LLM access proved counterproductive: relative to participants without any LLM access, untrained users wrote significantly shorter answers, committed more case misstatements, and scored marginally lower, though most differences fall short of conventional significance. Training reversed this pattern. Trained participants adopted the LLM at higher rates (41% vs. 26%; p = 0.044), scored 0.27 grade points higher than untrained users–roughly one fine grade–(p = 0.027), and stated applicable rules more accurately (p = 0.014).
Principal stratification analysis suggests training operates primarily through adoption rather than effectiveness–the adoption lower bound (1.06) exceeds the effectiveness upper bound (0.42) at strict mean dominance–though confidence intervals are wide. Training also shifted who adopted: top-quartile students went from 0% adoption to 42%.
More broadly, these findings challenge the view that GenAI primarily benefits lower-skilled workers: without training, higher-ability practitioners opt out while lower-ability users adopt but unproductively. Realizing GenAI’s productivity gains requires investment in both access and instruction.