Issue with TCP transport on Pitagora

Dear users,
this is to inform you that after the maintenance of September 22nd, a new issue emerged on Pitagora, affecting only the simulations asking the TCP transport for MPI communication. In these cases, your job may hang at initialization phase, if you are using more than one node for your simulations.

In general, the forcing of TCP transport is discouraged, and the default one selected by UCX should be used whenever possible, as it is typically more efficient and reliable. However, some applications require TCP transport to function correctly on PITAGORA, and are therefore affected by the problem.

It is not possible to return to the previous state because of a relevant network configuration that was necessarily performed on the September maintenance; however, it is possible to adjust your job to force MPI processes to use the correct network interface for TCP communication. The details can be found in a new “Known Issues” section at the userguide for Pitagora: https://docs.hpc.cineca.it/hpc/pitagora.html#known-issues-tcp

We apologize for the inconvenience.
Regards,
HPC User Support – CINECA