Or (less likely than Fran's suggestion), pushing it. It pretty much has to be some kind of lateral motion, as nothing else will pluck the string.

Try lifting the finger away from the fingerboard at slightly different angles (from the E side to the G side) and see if there's a spot where it goes away. If so, that's when you're lifting the finger straight up, even if it doesn't feel like it.

Also, the fingers should be curved. If you're having trouble with getting them that way, bring your left elbow further in (more in front of your body).  If any of that hurts, ease off, and work into it slowly. The "normal" position for holding the violin is definitely odd from your body's point of view.

If all else fails, shoot a video. Try and get closeups of your left hand from something halfway similar to how you see it. (Unless you can take your head off, getting a perfect shot probably won't happen. ) If whatever you shoot it with has a slo-mo feature (most phones do), shoot a version with that, too.

Another possibility just occurred to me. If you're getting your hand/fingers WAY over, you might actually be coming at the string from the other side, and when you lift your fingers (even straight up), you're plucking the string with the tip of your finger.  That would be a happy problem, because while it would mean you'd need to change how you're doing things, it would also mean that you're more limber than rubber tubing, and should be able to get your hand into any position needed.

Hope this helps