Always check if the factors make sense from a theoretical perspective and make sure to correctly interpret the different statistics (KMO / Bartlett / Kaiser's criterion / eigenvalues / explained variance / scree plot of the factor loadings to see if there are obvious inflexion points, etc.). Remember that software just runs a factor analysis and churns out some numbers - the interpretation of such an analysis is crucial and depends on theory. Lastly, if the two items are really forming a separate, fifth, factor, you should indeed remove the items from the analysis altogether: they are not a solid representation of any of the four constructs that the scale is supposed to represent. In addition, you could run a factor analysis that you force to converge on four factors, to see if those two items are indeed then grouped into a different factor. If the factor loadings of these two items are > 0.5 (a common threshold) with one of the other four factors and are not cross-loading with multiple factors (just those two), you may opt to make them part of such a factor instead and ignore the suggested fifth factor. There is not enough information to go on.

Do factor analysis in spss 21