As I can see in the logs you are doing right: You are using 64-bit players.
Perhaps it is madVR consuming a lot of resources. Try to select another renderer: EVR custom presenter for example.
To get correct colors with HDR content I recommend to use mpv player from SVP bundle:
SVP tray menu - Open file via mpv...