Human-AI Collaboration for Complex Decision-Making

With the rise of artificial intelligence (AI), companies are rushing to develop AI solutions for enhanced business decision-making. However, we often underestimate the human knowledge and involvement required to successfully implement such solutions. In complex and safety-critical applications, instead of having fully autonomous systems, we typically still have human domain experts make the final decisions, but increasingly assisted by advanced AI tools. For effective human-AI interaction, the most important requirement is trust. If the human decision-maker does not trust the AI model, they will not use it. In most of our applications of interest, the AI model is a complex optimization model, and the user is a domain expert but not an optimization expert. It is remarkable how often projects involving such models fail in practice for one of two reasons: (i) The model does not capture all factors that the human expert takes into account in their decision-making and hence arrives at solutions that are deemed unreasonable. (ii) The user does not understand the solution provided by the optimization model; for example, they cannot see the reasoning behind a solution that is different from what they expected. 

We tackle these important practical challenges by developing new systematic approaches that enable a human-in-the-loop decision-making framework as depicted at a very high level in the figure below. Here, the AI agent does not only suggest a solution but can also provide explanations to the human user, who in turn can query modifications or additional explanations. The AI agent can also use feedback from the user in the form of their queries and the decisions that they ultimately implemented to infer parts of the model that need further improvement to better align with human expert knowledge.

When we develop a new optimization model, we commonly see that its solutions do not align with the human expert’s decisions. One major reason for this is that the human expert has tacit knowledge about the problem, gained over many years of experience, that cannot be readily incorporated into the model. Usually, this is when the modeler and the user enter a tedious process of trial and error to tune the optimization model such that it provides reasonable solutions; we call this process model reconciliation as it, conceptually, reconciles the computational model with the human expert’s mental decision-making model. In our work, we have proposed a data-driven approach to model reconciliation, where the key idea is to model a human expert’s (or, more generally, an autonomous agent’s) decision-making process as a mathematical optimization problem. Then, given observed decisions made by the expert, we infer the expert's mental optimization model assuming that these decisions are optimal or near-optimal solutions to that model. In the operations research literature, this problem is referred to as data-driven inverse optimization. Inverse optimization gives rise to large-scale bilevel optimization problems that are very challenging to solve. In our work, we strive to contribute to the theoretical and algorithmic treatment of inverse optimization as well as demonstrate the practical impact it can have in real-world applications.

Explainability has recently become a major focus in machine learning research, where the goal is to explain the predictions from data-driven black-box models such as deep neural networks. Compared to most machine learning models, optimization models generally have a higher degree of interpretability as they consist of constraints that are typically derived explicitly from physical laws or logical expressions. However, although the model formulation may be easily interpretable, the process (or algorithm) of obtaining the optimal solution often is not. In fact, for all practical purposes, an optimization algorithm can be viewed as a black box. Especially in large-scale optimization involving many interacting variables and parameters, the complexity can be overwhelming and therefore limit the user’s ability to understand why the provided solution is optimal. Current explainability tools for optimization are either very application-specific or general but rather limited in their capability (e.g. simple what-if analysis and LP sensitivity). In our work, we aim to develop a portfolio of new explainable optimization approaches that provide effective contrastive explanations, identify causal relationships between interpretable features and decisions, and reduce model complexity for improved explainability.

• Lu, Y.-A., Hu, W.-S., Paulson, J. A., & Zhang, Q. (2025). BO4IO: A Bayesian optimization approach to inverse optimization with uncertainty quantification. Computers & Chemical Engineering, 192, 108859.
• Rathi, T., Gupta, R., Pinto, J. M., & Zhang, Q. (2024). Enhancing explainability of stochastic programming solutions via scenario and recourse reduction. Optimization & Engineering, 25, 795-820.
• Gupta, R. & Zhang, Q. (2023). Efficient learning of decision-making models: A penalty block coordinate descent algorithm for data-driven inverse optimization. Computers & Chemical Engineering, 170, 108123.
• Gupta, R. & Zhang, Q. (2022). Decomposition and adaptive sampling for data-driven inverse linear optimization. INFORMS Journal on Computing, 34(5), 2720–2735.