It sounds like the TheC64 stick is using a matrix, rather than a common ground. If that's the case, you'll get behavior similar to what you describe: controls won't activate correctly, or you'll get two (or more) controls activated with one button press.
A matrix input doesn't make sense for less than 4 inputs, but with with 6 or more inputs, you start seeing a cost savings in terms of pins. (6 inputs uses 5 pins, 9 inputs use 6 pins, and so on.)
I haven't looked inside the joystick that comes with the 64 Mini, but with 4 directional controls, a fire button, and at least 4 additional buttons, I wouldn't be surprised if it's using a matrix for input. In that case, you can't wire an Atari style DB9 to this stick; you'd probably be better off at that point trying to build a custom controller using a Teensy. (Something I plan to experiment with.)