PDEs & Applications Seminar
Tuesday, October 31st, 2023
Time: 9:30 a.m. Place: Jeffery Hall, Room 319
Speaker: Somnath Pradhan (Queen’s)
Title: Robustness to Incorrect Models and Discrete Approximations for Controlled Diffusions under Several Cost Criteria
Abstract: Typically, a system designer is given an approximate model for which policies are designed and then applied to a true model, leading to the problem of robustness to model mismatch. An additional related problem is on approximations of optimal control problems involving continuous space and time problems.
We first establish robustness of optimal policies under the discounted cost, cost up to an exit time, and ergodic cost with respect to functional perturbations involving controlled non-degenerate diffusions. Our approach builds on the regularity properties of optimality equations via a PDE theoretic analysis leading to a unified approach for several optimality criteria.
Then, we show that the costs are continuous on the space of stationary control policies when the policies are given a topology introduced by Borkar [V. S. Borkar, A topology for Markov controls, Applied Mathematics and Optimization 20 (1989), 55-62]. The same applies for finite horizon problems when the control policies are Markov, and the topology is revised to include time also as a parameter. We then establish that finite action/piecewise constant stationary policies are dense in the space of stationary Markov policies under this topology and the same holds for continuous policies. Using these, we establish that finite action/piecewise constant policies approximate optimal stationary policies with arbitrary precision.
This gives rise to the applicability of many numerical methods such as policy iteration and stochastic learning methods for discounted cost, cost up to an exit time, and ergodic cost optimal control problems in continuous-time. As a further utility, by showing additionally that continuous policies are dense in the space of stationary policies, we show that one can obtain a discrete-time Markov Decision Process whose solution (available via a rich collection of both analytical and simulation based methods) can be interpolated/extended for an original continuous-time problem under each of the criteria presented.
We will finally present some current research involving robustness to Brownian noise idealizations, as well as discrete-time approximations under general information structures involving partial information and decentralized information.
