When loading some specific binary PLY, the textures displayed are wrong. It seems to use the texture file (which is weird btw) colors but not the correct position. But it's fine in other 3D viewer ...

ply files do not support textures theoretically. Since the standard does not provide a way to attach them, each importer/exporter goes on its own. Could you please provide an example ply file exported ...