Is the camera being 'dragged' at all?
Camera Class | Microsoft.Samples.DirectX.UtilityToolkit Namespace