Jaap -- much of the modern "TV" 75 ohm coax these days has a foam dielectric (with a Velocity Factor ~ 0.75) instead of the classic RG-59 (with V.F. ~0.66). If you cut the coax by the standard formulae, this could be the problem.
Using the coax as a quarter-wave 100 -> 50 ohm transformer is not very sensitive. What I do with my MFJ269 is to measure the frequency where an open on the far end looks close to being a short and where a short on the far end looks like an open. You should be able to measure this to ~1 MHz or better. Even better, do this test with the 3/4 wave piece, where you will get a lever-arm three times longer. I have found that commercial coax varies by 2-3% from lot-to-lot so I always test the coax each time I build a new "rat-race" hybrid.
Hope this helps -- 73, Tom