Deep reinforcement learning for automated reinforced concrete structural design

Computer-aided civil structure design methods have been investigated for decades with various optimization techniques and neural networks. This study proposes a novel concept to aid structural design procedures using a reinforcement learning (RL) based approach, with a particular focus on the reinforced concrete (RC) beam design as a case study. We trained an RL agent called Deep deterministic policy gradient (DDPG) with a convolutional neural network (CNN) function approximators. The agent was successfully trained to design simply supported RC beams subject to the American concrete institute (ACI 318) provision without any data-driven supervision. A python-based RC beam design environment was developed and used to simulate RC beam designs with custom reward functions that encourage the agent to minimize the cost by maximizing the reward. The DDPG agent self-learned the cost-effective RC beam design through the 100,000 randomly generated design cases during the training procedure. The trained agent is able to design the RC beam in a cost-effective way with consideration of flexural and shear reinforcement arrangements. In addition, it generates near-optimal design parameters without iterative procedures under various design conditions. The performance of the agent was validated with 100 design cases and comparative studies. The proposed method showed great promise in cost-effective and automated RC beam design.

The RC beam design task is modeled as a DRL problem with a state, action, and reward format. In the first step, the agent observed the initial state, which was given with the design requirement, and generated a design parameter such as sectional dimension, rebar size, and quantity. Then, the environment generated a simply supported beam design based on ACI-318, and the defined reward functions calculated the corresponding reward; when a design failed to satisfy the design code (i.e., ACI-318) requirements, the negative value of the reward was generated; when a design succeeded, the positive value of the reward was produced. The generated design parameters and reward were observed by the agent again, and the agent generated another design parameter.

The below figures showing the training and test result using the proposed custom environment and reward strategy. The agent successfully trained under 100,000 randomly generated design cases. The performance of the trained agent was validated using 100 randomly generated design cases. The trained agent was able to design RC beam without code violation but designed in near-optimal material cost. The bottom figure shows an example of complete RC beam designed by AI agent.

For the extended demonstration with a more complicated example case, the same RL based beam design strategy was implemented to a two dimensional 3-story and 3-bay RC frame structure. The example RC frame structure has been previously studied by two research groups using optimization algorithms; The genetic algorithm (GA) (Lee & Ahn, 2003) and the big bang big crunch (BB-BC) algorithm (Kaveh & Sabzi, 2012). In this study, only the beam portion of the frame structure was considered, as the column design is beyond the scope of this study.

Table 1. Designs comparison for beams in three-bay, three story reinforced concrete frame structure

The agent was retrained using new material cost, load factor, and moment equation using the same strategy. The beam dimensions and main rebar for negative moment were decided using Mu.neg by RL agent, and the rebar for a positive moment was arranged using ρ equation. The design results are summarized in Table. As shown in Table, the RL-based design approach was able to design the continuous beams close to the optimized design by GA and BB-BC algorithms. The beam material cost of the agent-generated design was a little bit higher but closed to optimized results. However, the number of required analyses from proposed RL method had more than > x1000 computational efficiency comparing to the optimization methods.`

This work is published in the Computer‐Aided Civil and Infrastructure Engineering(CACAIE), wiley.

Jeong, J. H., & Jo, H. (2021). Deep reinforcement learning for automated design of reinforced concrete structures. Computer‐Aided Civil and Infrastructure Engineering, 36(12), 1508-1529. (IF: 11.775) Top rank journal (1 out of 136 journals) within Civil Engineering Click here!

Reference

[1] Lee, C., & J.Ahn. (2003). Flexural design of reinforced concrete frames using a genetic algorithm. Journal of Structural Engineering, 129(1), 105–115

[2] Kaveh, A., & Sabzi, O. (2012). Optimal design of reinforced concrete frames Using big bang-big crunch algorithm. International Journal of Civil Engineering, 10(3), 189–200