Hi Tim!
The best codec regarding quality is always no codec!
If you have the memory space and time for copying I would recommend using uncompressed AVI. This would provide the best quality due to the fact, that you can't have any artifacts or color conversions. All less reduced codecs are fine as source video format.
Apple ProRes is as well fine as PhotoJPEG with high quality. The animation codec can be used as well.
I am not sure if ProRes can handle this huge pixel size (4000x4000) as I am not really familiar with it.
As you already said yourself, mxl-files are a proprietary format of coolux. That's why you can only encode these files with our inbuilt Encoder Extension within Pandoras Box Software.
We need to have this file-type because MPEG-files are only standard-compliant up to a resolution of 1920x1152px at 60Hz in HighProfile and HighLevel. Due to this all larger resolutions have to be in another container-format.
Regards,
Markus