The Geolink manual states that you can assign multiple functions to the same channel.

I have channel 10 which is on a 3 position switch, and I'm trying to assign it this way
Pos 1 (up) Geolink OFF
Pos 2 (Center) Position Hold
Pos 3 (Down) Return to Home

I have tried every curve possible and every combination, but no luck. I can see that Return to home can have 4 different states (Normal return, Programming, Programming-all banks and Return to power on spot), But I can't figure out how to separate it from the position hold.

I can have both off, and I can have the both on, or I can have the Position hold on, and any one of the (non- normal) return states on, but I can't figure out a combination where I can have the Position hold enabled and the Return to Home off.

Has anyone tried this? How did you get it working?

I think you have to use at least two channels driven by the same switch
and use some mixes or conditions to have this result

Yes, except that is not what the manual says the system is capable of.

3.1.2 Dedicated GeoLink channel [Recommended]

More strightforward way is to assign a GeoLink feature to any free channel in the General/Channels window. In the right part of the Spirit Settings software select which feature you would like to assign. (for example F: GeoLink Return to Home). Then select appropriate channel which will be used for engaging. You can use same channel for multiple features at the same time.

I want to use only one channel (in this case Ch.10), assign it to one 3 position switch, then each switch position trigger a unique Geolink function. The toggle has 3 value states: -100%, 0%, +100%. Those values can be changed with a curve to any arbitrary value, but the idea is that each assigned function is only triggered when the associated value is present (eg. -100% = All OFF or alternately Geo Fence ON | 0% = Location Hold ON, Return Home OFF | +100% Location Hold OFF, Return Home ON.

The system of course does what is written in the manual, maybe just not the way you wish.

I am currently using:
one channel to enable altitude hold together with position hold
one channel to enable altitude limit together with geo fence

The way you wish to use it, need some kind of offset on each geolink function,
because altitude limit / geo-fence / position and altitude hold
all enable when the assigned channel is at +100

Someone asked to Thomas to implement the same way that is used on betaflight
but don't know if that will be implemented and when.

EDIT: ( found the topic )
viewtopic.php?f=21&t=5447&p=38865

I was able to achieve was I was looking for by assigning a second channel to the same switch, with the switch position activating the channel, but ultimately, that's just a workaround.

Hopefully Thomas can chime in once he gets back from his vacation. It's fortunate that I had enough available channels, but if this would have been a more complex models (with retracts and lights, smoke and other bells and whistles, I wouldn't have been able to pull it off.