In this study on music learning, we develop an average reward based adaptive parameterisation for reinforcement learning meta-parameters. These are tested using an approximation of user feedback based on the goal of learning the nursery rhymes Twinkle Twinkle Little Star and Mary Had a Little Lamb. We show that a large reduction in learning times can be achieved through a combination of adaptive parameters and random restarts to ensure policy convergence.