Modelling and stability analysis in human population genetics with selection and mutation