This paper presents improved automatic moving target detection and tracking framework that is suitable for UAV imagery. The framework is comprised of motion compensation phase to detect moving targets from a moving camera, target state estimation with Kalman filter, and overlap-rate-based data association. Finally, P-N learning is used to maintain target appearance by utilizing novel structural constraints to select positive and negative samples, where data association decisions are used as positive (P) constraints. After learning target appearance, a cascaded classifier is employed to detect