We were using MPI for communicating between nodes in a linux cluster. We replaced it with JGroups because it's more flexible, more stable, has better failure modes, and was plenty fast and network bandwith-friendly. The system we built got used in a critical production environment and performed wonderfully for many months.
It should be noted this is a toolkit, not an end-user product. It's supposed to be used to build your own multi-server application. Don't expect GUIs and other end-user amenities. This is a powerful, stable tool for developers to use to build those things.