Developers decide whether to trust software by reading evidence such as a passing test suite, a fuzzing campaign that finds no new crashes, or a dynamic analysis that reports no vulnerable data flows.
I study when that evidence is reliable, and I build tools that make it more reliable.
Much of this work targets the Java Virtual Machine and open-source projects that test every change in continuous integration (CI).
I release my tools and datasets under open-source licenses, and I have contributed changes based on my findings to widely used testing tools, including Maven Surefire and Pitest.
Coding agents now write a growing share of code, often along with the tests that are supposed to check it.
As writing code gets cheaper, the harder problem is deciding whether to accept what an agent produced, and most of my current work addresses that problem.
Acceptance evidence for agent-written code: An agent that can observe its acceptance checks will fit its output to them.
I study this problem in agent-driven migrations of legacy software, where the original program defines the expected behavior but the port may deliberately depart from it, for example to remove memory safety errors.
In a case study that I presented at the 2026 Workshop on Software Systems across the National Laboratories, coding agents ported a 280 KLoC C program to JavaScript.
They built recorders, input generators, and differential tests that compare the port with the original, and direct checks confirmed that these tools worked.
They also wrote differential tests that passed while observing no meaningful program state.
I argue that accepting such a port requires a certificate of bounded agreement with intent.
A domain expert chooses which projection of program state must agree with the original, and within what tolerance, and testing supplies a confidence bound over a stated input distribution.
Agents can now build most of that testing apparatus cheaply.
People still have to decide what it observes.
My ongoing work applies these certificates to more legacy systems and develops ways to detect tests that pass without checking anything.
Mutation testing is one way to check the checks, because it measures whether a test suite detects small, deliberate changes to the program.
LLMorpheus, built with Frank Tip and Max Schäfer, prompts a large language model to propose mutants, and it produces mutants resembling real bugs that StrykerJS, a conventional mutation testing tool, cannot generate (Tip, Bell, and Schäfer, 2025).
I also study how developers use AI tools in practice.
We compared developers’ experiences with ChatGPT and Stack Overflow through a thematic analysis of more than 1,700 Reddit posts (Li, Mynatt, Mishra, and Bell, 2025).
With Arjun Guha’s group, I helped build AgentPack, a dataset of 1.8 million code edits co-authored by coding agents and humans in public GitHub repositories.
Flaky tests: Flaky tests pass and fail on the same version of the code, so a failure in CI may or may not indicate a bug.
My earliest work in this area studied tests that depend on the order in which they run (Bell and Kaiser, 2014; Bell, Kaiser, Melski, and Dattatreya, 2015; Gambi, Bell, and Zeller, 2018).
DeFlaker identifies flaky failures without rerunning tests by checking whether the failing test executed any recently changed code (Bell, Legunsen, Hilton, Eloussi, Yung, and Marinov, 2018).
To learn how tests become flaky, we built “software archaeology” experiments that examined thousands of revisions of open-source projects, repeatedly running each test suite to find and profile flaky tests (Lam, Winter, Wei, Xie, Marinov, and Bell, 2020).
For FlakeFlagger, we reran test suites 10,000 times and found that fewer than half of the flaky tests appeared within the first 100 runs (Alshammari, Morris, Hilton, and Bell, 2021).
The FlakeFlagger dataset helped start a line of research on predicting flaky tests with machine learning.
More recently, we found that 46.5% of the flaky tests in a study of 52 projects fail depending on the computational resources available when they run (Silva et al., 2024).
FlakeRake reproduces rare timing-dependent failures by inserting targeted sleep calls (Rahman, Massey, Lam, Shi, and Bell, 2024).
In a study of 230,439 failure messages, we measured when matching a new failure against previously seen failures can safely label the new failure as flaky (Alshammari, Ammann, Hilton, and Bell, 2024).
A 2026 preprint with colleagues at Heidelberg University and SAP shows that the strong benchmark scores reported for code-based flakiness detectors depend on labeling shortcuts and evaluation protocols.
It argues for reframing flakiness prediction around whether an observed failure is flaky and how likely a test is to fail in a given execution environment.
My ongoing work uses coding agents to repair flaky tests and submits each repair upstream as a pull request.
Fuzzing and dynamic analysis: Fuzzers generate inputs to reveal untested behavior, including crashes and security vulnerabilities.
Many of my tools build on dynamic taint tracking, which follows data from a program’s inputs as the program runs.
Phosphor brought precise taint tracking to unmodified JVMs (Bell and Kaiser, 2014), and Conflux extended it to information that flows through control dependencies (Hough and Bell, 2021).
Galette, which Katherine Hough and I built, has since replaced Phosphor.
It propagated taint tags with perfect accuracy on 3,451 benchmark programs across all four long-term support (LTS) versions of Java (Hough and Bell, 2025).
Rivulet uses these analyses to amplify developers’ existing tests and detect code injection vulnerabilities (Hough, Welearegai, Hammer, and Bell, 2020), and CONFETTI combines concolic execution with generator-based fuzzing (Kukucka, Pina, Ammann, and Bell, 2022).
Fuzzer design decisions are rarely evaluated on their own, so we study them directly.
Examples include how to perform crossover in parametric fuzzing (Hough and Bell, 2024) and how the individual mutation operators in AFL++ affect coverage and bug detection (Kukucka, Pina, Ammann, and Bell, 2024).
Software supply chain: Applications depend on large numbers of open-source packages, and a package manager’s dependency solver chooses which version of each package to install.
We formalized the semantics of dependency solving and built MaxNPM, a drop-in replacement for NPM that lets developers combine goals such as choosing newer versions, avoiding duplicate packages, and reducing known vulnerabilities (Pinckney, Cassano, Guha, Bell, Culo, and Gamblin, 2023).
To study how updates propagate, we built npm-follower, a 20 TB dataset of every version of every package on NPM (Pinckney, Cassano, Guha, and Bell, 2023).
With it, we found that security patches reach downstream packages quickly in about 90% of cases, and that the remaining delays come from developers’ imperfect use of semantic versioning (Pinckney, Cassano, Guha, and Bell, 2023).
Interpretability infrastructure: I am a co-PI of the National Deep Inference Fabric (NDIF), an NSF-funded service that gives researchers remote access to the internals of large open-weight language models through the NNsight library (Fiotto-Kaufman et al., 2025).
I oversee NDIF’s open-source community engagement and engineering processes, and with David Bau I lead Workbench, a web application for exploring a model’s internals and running interpretability experiments without writing code.
With my PhD student Gwen Lincroft, I study how people without interpretability expertise use Workbench to inspect a model and intervene on its internals.
Testing education: Mutation testing can also give students feedback on the tests that they write.
Across 2,711 student submissions, mutation score correlated strongly with detection of instructor-seeded faults (Perretta, DeOrio, Guha, and Bell, 2022).
In a controlled experiment with 4,122 students, instructor-written hints about undetected mutants produced a small but statistically significant improvement in student test suites (Perretta, DeOrio, Guha, and Bell, 2025).
In CS 3100, we pair each curated mutant with an instructor-approved hint that a language model expands on request.
When expansions were available on demand, 46% of students requested at least one.
Each expansion cost about three cents (Williams Earle and Bell, 2026).
References
Curated Semantic Mutants: Multi-purpose Artifacts for Grading and Hinting Student Test Suites. Rebecca Williams Earle, Jonathan Bell. Proceedings of the 2026 ACM SIGPLAN International Symposium on SPLASH-E, 2026 [pdf].
DetailsDynamic Taint Tracking for Modern Java Virtual Machines. Katherine Hough, Jonathan Bell. Proceedings of the ACM on Software Engineering. 2025 ; 2(FSE). [pdf]. [artifact].
DetailsLLMorpheus: Mutation Testing using Large Language Models. Frank Tip, Jonathan Bell, Max Schäfer. IEEE Transactions on Software Engineering. 2025 : 1–21. [pdf]. [artifact].
DetailsNNsight and NDIF: Democratizing Access to Open-Weight Foundation Model Internals. Jaden Fried Fiotto-Kaufman, Alexander Russell Loftus, Eric Todd, Jannik Brinkmann, Koyena Pal, Dmitrii Troitskii, Michael Ripa, Adam Belfki, Can Rager, Caden Juang, Aaron Mueller, Samuel Marks, Arnab Sen Sharma, Francesca Lucchetti, Nikhil Prakash, Carla E. Brodley, Arjun Guha, Jonathan Bell, Byron C Wallace, David Bau. The Thirteenth International Conference on Learning Representations, 2025 [pdf].
Details“Always Nice and Confident, Sometimes wrong”: Developer’s Experiences
Engaging Generative AI Chatbots Versus Human-Powered Q&A
Platforms. Jiachen Li, Elizabeth D. Mynatt, Varun Mishra, Jonathan Bell. Proceedings of The 28th ACM SIGCHI Conference on Computer-Supported Cooperative Work & Social Computing, 2025 [pdf].
DetailsInstructor-Written Hints as Automated Test Suite Quality Feedback. James Perretta, Andrew DeOrio, Arjun Guha, Jonathan Bell. Proceedings of the 56th ACM Technical Symposium on Computer Science Education, 2025 [pdf].
DetailsThe Effects of Computational Resources on Flaky Tests. Denini Silva, Martin Gruber, Satyajit Gokhale, Ellen Arteca, Alexi Turcotte, Marcelo d’Amorim, Wing Lam, Stefan Winter, Jonathan Bell. IEEE Transactions on Software Engineering. 2024 : 1–18. [pdf]. [artifact].
DetailsAn Empirical Examination of Fuzzer Mutator Performance. James Kukucka, Luís Pina, Paul Ammann, Jonathan Bell. Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, 2024 [pdf]. [artifact].
Details230,439 Test Failures Later: An Empirical Evaluation of Flaky Failure Classifiers. Abdulrahman Alshammari, Paul Ammann, Michael Hilton, Jonathan Bell. Proceedings of the 2024 IEEE Conference on Software Testing, Validation and Verification, 2024 [pdf]. [artifact].
DetailsAutomatically Reproducing Timing-Dependent Flaky-Test Failures. Shanto Rahman, Aaron Massey, Wing Lam, August Shi, Jonathan Bell. Proceedings of the 2024 IEEE Conference on Software Testing, Validation and Verification, 2024 [pdf]. [artifact].
DetailsCrossover in Parametric Fuzzing. Katherine Hough, Jonathan Bell. Proceedings of the 2024 International Conference on Software Engineering, 2024 [pdf]. [artifact].
DetailsFlexible and Optimal Dependency Management via Max-SMT. Donald Pinckney, Federico Cassano, Arjun Guha, Jonathan Bell, Massimiliano Culo, Todd Gamblin. Proceedings of the 2023 International Conference on Software Engineering, 2023 [pdf]. [artifact].
DetailsA Large Scale Analysis of Semantic Versioning in NPM. Donald Pinckney, Federico Cassano, Arjun Guha, Jonathan Bell. Proceedings of the 20th International Conference on Mining Software Repositories, 2023 [pdf]. [artifact].
Detailsnpm-follower: A Complete Dataset Tracking the NPM Ecosystem. Donald Pinckney, Federico Cassano, Arjun Guha, Jonathan Bell. Proceedings of the 2023 ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2023 [pdf]. [artifact].
DetailsOn the Use of Mutation Analysis for Evaluating Student Test Suite Quality. James Perretta, Andrew DeOrio, Arjun Guha, Jonathan Bell. Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis, 2022 [pdf]. [artifact].
DetailsCONFETTI: Amplifying Concolic Guidance for Fuzzers. James Kukucka, Luis Pina, Paul Ammann, Jonathan Bell. Proceedings of the 2022 International Conference on Software Engineering, 2022 [pdf]. [artifact]. [code/git].
DetailsA Practical Approach for Dynamic Taint Tracking with Control-flow Relationships. Katherine Hough, Jonathan Bell. ACM Transactions on Software Engineering and Methodology. 2021 ; 31(2). [pdf]. [code/git].
DetailsFlakeFlagger: Predicting Flakiness Without Rerunning Tests. Abdulrahman Alshammari, Christopher Morris, Michael Hilton, Jonathan Bell. Proceedings of the 2021 International Conference on Software Engineering, 2021 [pdf]. [artifact]. [code/git].
DetailsA Large-Scale Longitudinal Study of Flaky Tests. Wing Lam, Stefan Winter, Anjiang Wei, Tao Xie, Darko Marinov, Jonathan Bell. Proceedings of the ACM on Programming Languages. 2020 ; 3(OOPSLA). [pdf]. [artifact].
DetailsRevealing Injection Vulnerabilities by Leveraging Existing Tests. Katherine Hough, Gere Welearegai, Christian Hammer, Jonathan Bell. Proceedings of the 2020 International Conference on Software Engineering, 2020 [pdf]. [artifact]. [code/git].
DetailsPractical Test Dependency Detection. Alessio Gambi, Jonathan Bell, Andreas Zeller. Proceedings of the 2018 IEEE Conference on Software Testing, Validation and Verification, 2018 [pdf]. [code/git].
DetailsDeFlaker: Automatically Detecting Flaky Tests. Jonathan Bell, Owolabi Legunsen, Michael Hilton, Lamyaa Eloussi, Tifany Yung, Darko Marinov. Proceedings of the 2018 International Conference on Software Engineering, 2018 [pdf]. [code/git].
DetailsEfficient Dependency Detection for Safe Java Test Acceleration. Jonathan Bell, Gail Kaiser, Eric Melski, Mohan Dattatreya. Proceedings of the 10th Join Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering, 2015 [pdf].
DetailsPhosphor: Illuminating Dynamic Data Flow in Off-The Shelf JVMs. Jonathan Bell, Gail Kaiser. Proceeding of the 29th ACM SIGPLAN Conference on Object Oriented Programming Systems Languages and Applications, 2014 [pdf]. [artifact]. [code/git].
DetailsUnit Test Virtualization with VMVM. Jonathan Bell, Gail Kaiser. Proceedings of the 2014 International Conference on Software Engineering, 2014 [pdf]. [code/git].
Details
A complete list of my publications is also available.