A new cintool is available: cinhak

Dear users,

A new tool called cinhak is now available in the “cintools” module, which is loaded by default upon login.

This is a simple wrapper designed to help avoid wasting computational budget by detecting application hangs that Slurm does not automatically terminate.

The tool monitors a user-specified file that reflects the application’s progress. If the file remains unchanged for a configurable period (10 minutes by default), the application is assumed to be hung and the corresponding job is automatically terminated. This approach is particularly useful for long-running workloads where a stalled application may continue consuming resources without making any progress.

To use it, simply prefix your command with cinhak --filename=<progress-tracking-file>. For example:

cinhak --filename=progress.out mpirun ./mpi_hello_world.exe > progress.out

You can view all available options and their default values by running:

cinhak --help

For more information, please refer to the “How can I detect and kill a hanging job automatically” section in our user guide.

Best regards,
HPC User Support @ CINECA