HDR performance hit?

Got the HDR mod working with the Witcher 3 yesterday and it looks great. However, my FPS dropped from an average of 75 fps at 4k to 55. Is this normal? Would really like to remain above 60 if possible.

I think you can disable the 8-bit remastering if it is enabled, to save some performance.