Check your scrub angle, basically the futher you move the wheel out from the axle, the harder it is to steer. I set mine up so the botom of the king pin bolt is directly over the inside lip of the rim. (I machine the head down to about 1/4th of the thickeness of the head) this still gives plenty of room for weight jacking.
Its simple levers, going outward (rim) makes it harder. You can fix this by adding plates to where your heim joints bolt to, there by extending your axle width, then remove or change your axle spacers so the rim can come in.