Development of Human Muscle Control Framework Based on Deep Reinforcement Learning

Computational human body models (HBMs) are important tools used in biomechanics research to predict human responses under external loads. To improve the predictive capabilities of HBMs in some loading scenarios, it becomes important to incorporate a motor control mechanism that affects the human response by generating active muscle forces around skeletal joints. Current efforts to integrate muscle control into HBMs rely on feedback-based controllers which have been precisely tuned for specific load cases and may not generalize to cases beyond which it has been tuned. The dissertation proposes a novel approach to muscle control based on deep reinforcement learning (RL).

Reinforcement learning algorithms are recent advancements in the field of machine learning and allow for a complex system to learn how to work successfully to achieve the desired outcome and is analogous to how a child learns to walk by trying over and over again by constantly interacting with the surrounding environment. The proposed research presents a comprehensive study on the design of RL Muscle Activation Control (RLMAC) using detailed musculoskeletal multibody (MB) models and volunteer testing data. The central goal of this dissertation was to evaluate the utility of RL-based algorithms in muscle control for generating voluntary kinematics with eventual application to complex external loadings. The dissertation also examines the application of the trained muscle controller to changes in anthropometry, the addition of external mass to the body (such as helmets), and changes to the external loading environments.

An initial proof-of-concept study on the use of RLMAC was performed using a multibody model of the human arm incorporating muscles responsible for motion about the elbow. The human arm model provided a simple model setup with a revolute joint at the elbow which made it convenient for the preliminary analysis. The RLMAC was trained to perform extension and flexion movement of the lower arm by activating the muscles, and the trained controller could generate goal-directed arm movements, synthesize the same motion in the presence of an external force field, and the trained controller could also maintain the stability of the elbow joint to high magnitude impulse loads. Following the initial investigation, the RLMAC was integrated into the head and neck body region MB model with the anatomy of a 50th percentile male. The presence of multiple non-linear joints and the complex muscle orientation of the cervical spine make the head-neck complex a suitable body region to evaluate the ability of RL-based controls to generate coordinated muscle forces for joint control. At first, the RLMAC was trained assuming symmetry about the sagittal plane (i.e., left and right muscles were assigned identical activations). With the symmetrical control model, the RLMAC architecture was developed to maintain stability under gravity and synthesize the extension-flexion motion of the neck. The same control approach was then extended to all DOF models where each muscle was activated individually to maintain the desired posture.

A series of volunteer tests were performed to finetune and calibrate the architecture of the RL controller. The test subjects were asked to perform fast goal-directed rotations of the head in the sagittal plane (extension-flexion) and transverse plane (axial rotation), and the data gathered from the volunteers were used as datasets for the model validation. The trained RLMAC could replicate the desired head movements with both the symmetry model and the all-DOF model.

Finally, range of applicability studies were performed to gauge the ability of the RL controller to adapt to novel scenarios and develop responses to external loads, for which it has not been explicitly trained. The trained RLMAC was able to adapt to changes in anthropometry and was also able to maintain stability with an added mass representing a helmet. The trained model could also react to impact loads which provide evidence of its potential for controlling HBMs under novel loading environments for which it has not been previously trained.

The dissertation provides a detailed insight into the development of general use HBM muscle controller with the capability to simulate commonly encountered chaotic scenarios. The proposed approach can be extended to other body regions as well with the eventual application at the whole-body level. Active HBMs will serve as important tools for the development of improved injury mitigation devices by accurately predicting the response and thus, the injury risks. Furthermore, the RLMAC framework can also be used in biomechanics applications such as gait and occupational health research.