One dreaded Friday morning, we started getting reports of multiple users in VMware Horizon getting kicked out of their desktops and even more unable to launch into their desktops with a Tunnel Session Not Created error. This was in an old poorly architected VMware Horizon environment so unfortunately, Security Servers were in play for tunneling external sessions. In this day and age, you likely don’t use Security Servers but if you do and you face this problem, read on!
When troubleshooting, we could immediately see those experiencing the problem were all external. Internal users didn’t have an issue. Makes sense! Only external users tunnel through the Security Servers.
There were no hints in the event logs or VMware log bundles to help us figure out where exactly the break was and how to fix it but it had to be the Security Servers. We discovered that there was an Edge firewall change that night and it seemed the obvious culprit. Security Servers by their nature live in the DMZ and of course could be impacted by such a change.
We were very fortunate to work with a great support engineer at VMware. When all of our attempts came up short, we called them. The engineer suggested a re-install and walked us through it. This was after over 10 hours of trying anything and everything else before calling. We contemplated a new install but though it was the absolute last resort.
Here is the conclusion:
Cause: At the time our belief was that the server was out of commission for so long that something went out of sync and didn’t pick back up when the change was complete.
However, Sean Massey who is an authority on Horizon suggested the firewall change broke the IPSEC tunnels between the Security Servers and the Connection Servers and in this case a re-install is the only fix. This makes a lot of sense, the engineer did not tell us that could be the cause but he did seem pretty confident it would fix it.
Fix: In this case, we ended up uninstalling VMware Security Server and re-installed it.
Maybe this will save somebody else 10+ hours 🙂
