关键词: 梯级水库随机优化调度, 深度强化学习, 深度Q网络算法, 时序差分思想, 探索利用策略

Abstract: Compared with a single reservoir, cascade reservoirs operation features a state space increasing exponentially. This paper describes a Deep Q-network (DQN) algorithm for deep reinforcement learning to solve the dimension disaster problem that is faced by the table-based reinforcement learning method in optimizing the long-term operation of cascade reservoirs. First, we derive a joint distribution function of stochastic inflow runoffs of the reservoirs based on the Copula function. Then, following the idea of time series difference, we construct a target neural network and a main neural network for approximating the values of the current action state and the next action state, respectively, and use ε-greedy algorithm to obtain optimal operation policy. Finally, the main parameters of reservoir operation are optimized by step to ensure operation efficiency. Compared with the Q-learning algorithm or its modification, the DQN algorithm improves the objective value of optimal scheduling, accelerates convergence, and avoids dimension disaster effectively in the optimization of cascade reservoirs operation.

Key words: stochastic optimal operation of cascade reservoirs, deep reinforcement learning, Deep Q-network algorithm, temporal difference idea, exploration and exploitation strategy