it would send CW when in SSB mode using the up/down buttons on the Heil.
Menu item 36 must be set to ON on the 817 TX rig AND the Keyer must be turned on... to do that, hit 'F', then spin the SEL knob to [VOX, BK, KYR] and make sure a black Arrow icon to the right of 'KYR' is there. If not, press the 'C' button to activate the Keyer. - If these are set correctly and the +/- buttons still do not activate the CW keyer... it sounds like a sign of a ground problem between the headset and the RJ-45 jack, which is what I think is going on with the lack of PTT when the speaker audio is not plugged in. (See below.)
It will not key the transmitter of either FT-817 unless the headset speakers are also connected to the same FT-817.
My Heil Traveler Dual will key my 817 without the headset speaker audio plug connected. I would suspect a ground problem going to the RJ-45 mic plug. Check continuity of the ground line between the 8 pin round plug and the RJ-45 plug... there are TWO Grounds and they are SEPARATE in the Heil adapter cable! Mic Ground is for the mic only and Ground is for the switches. - You will likely need to find a piece of very small gauge wire to make contact with the RJ45 contacts as a typical VOM probe will not fit in those small grooves. Unless you have a spare, unconnected RJ-45 jack handy... even better!
Heil Traveler Pinout: 1 Mic, 2 NC, 3 + Switch, 4 – Switch, 5 PTT Switch, 6 Ground, 7 Mic GND, 8 Earphone -- The outer metal shell of the 8 pin plug is not connected to ground, and should not show a connection to any of the pins. The +, - and PTT switches will make connection between their respective pins and ground (pin 6) when closed. The pin numbers should be visible on the female side of the 8 pin connection, visually transpose those locations for the male side.
For the 817 mic jack pinout, you can consult the 817 manual (page 5 in my 817ND manual). If you don't have one, the order is: Fast, GND, PTT, Mic, Mic GND, +5v, Up(+ switch), Down(- switch) However the direction that order goes depends on which way you are looking at the jack/plug... if I have the plug laying contacts up & clip down with the end facing me (cable running away) that would be the L-R order. If turned around, it would be R-L... you should be able to figure it out. =^)
You might check them all, but you want to make sure that the Heil Pin 6 goes to the RJ45 ground and should also show continuity to the speaker plug ground. (Also check that Heil pin 7 goes to mic ground on the RJ45 and that pin ONLY... it should not have continuity with the other ground connections; while unplugged, of course.) If these are not all correct, you have likely found your problem... Because it sounds like to me, you get PTT return through the ground connection of the speaker audio plug. When you disconnect that and there is no connection between Pin 6 and the appropriate RJ45 contact, no ground for either the PTT OR for the up/down buttons! (as referenced above)
If the above still doesn't locate the 'lack of PTT' then the next step would be to check the mic jack contacts in the radios. But if you DO determine that the cable is missing a proper Pin 6 ground, then it means a new adapter, a new RJ45 plug, or in a pinch, you might just ground the two radios together. (not the best practice, but it'll do in a pinch) - Which this all gets me thinking... quick short cuts... do you have another Heil Traveler adapter for RJ45 yaesu? try swapping them. Also, might plug up just the RJ45 part of the adapter and check pin 6 to radio case ground (BNC jack shield on front). If that's good, it's something else.
If you need any more help or clarification, include my offlist e-mail address to be sure and get my attention quicker. 73, Kevin N4UFO