If rudder is kicking to such extent (more than 45°) then it is possible, that the tail is not able to mechanically handle the load.
This mean, that whatever you will set, it will not help.
Can you tell me what is angle between the tail blades when you set it fully to one side?
If your tail limits are maxed (travel is fully used), then you have options:
- increase RPM (I guess that you will need to increase a lot since it does not work even in transition to inverted flight)
- put a longer tail blades
It is possible that the tail ratio is making the trouble. Can I know reason for gear ratio change from the stock?
Alternatively there is also a chance that what might cause problems are servos. Unfortunately I have seen many troubles with RJX/BK servos due to different issues
But this was usually with 700X series. When they are good, it is working exceptionally well.
It can be good to try to lower BEC voltage to 6V if possible (just a try) - this often helped with BK servos that were loosing its holding power.
You might also start the motor without blades to verify if nothing strange is happening. Also you can try to touch the servo cases by finger.
Regarding setup - For BLS servos you will need to increase Pirouette Consistency by 10-20 from normal (to e.g. 185 - 200).
Rudder Delay to 0.
Elevator Filter probably 3-4.
Nothing else should be that important.