Movement in that diresction can be caused by a lot of things, most not goood. Something is out of alignment, could be the relationship of the cam being "square" to the crank, could be the crank not perfectly aligned straight (you would be amazed on the tolerance level from the factory. The lifters could be out of alignment with the cam. Most of this is fixable, you just have to find it.
99% chance it was like this from the factory, core shift in the casting process or bad machining. You just have to start looking or band aid it like shims or they make thin flat roller bearing you might put behind the cam. There should be very little float on it, look for wear wear behind the cam gear.
Good luck.