Friday, January 23, 2009

What to do when your connectivity fails

We had a long, hard look at whether to change to the new ASP model run by eHealth Ontario--eHO--(previously called Smart Systems for Health Ontario, or SSHA), or to continue with our server, based at the hospital.

We decided to continue with our server for now.

There are several reasons for that decision. While I strongly believe that ASP is the way to go over the long term, I am reluctant to switch today. We now have good IT support for the server, through our own IT manager, and have bought additional hardware insurance. The server generally functions well and is closely monitored.

I cannot say the same for our connectivity, which is managed by eHO. We have had several outages in the past six weeks, and two outages this week. On Tuesday, we were off for the whole morning, as the eHO lines were down. Regular Internet was fine, so I was running on my backup line.

Trying to run a backup Internet line using the rules imposed by eHO is a daunting task. We have to use a SOFA (Small Office Firewall Appliance), through which the internet line connects to the office. I have a second, private, internet line coming to my office in case of failure. I had a private company install a failover router, so that the second line picks up when the first one fails. That itself failed the fist time SSHA's internet line went down. When SSHA was back up, the private modem interfered with the SSHA modem, so we could not re-connect properly.

What I had to then figure was how to force the two system to cooperate (a bit like what is happening between all the different players in the EMR field).

Basically, when we fail to connect to our server, we check Google. If that fails, then we know that it is the eHO lines (again). We go to the back closet, turn off the eHO modem and turn on the private internet modem. The private modem bypasses the SOFA, and is connected straight to the main router for my office.

What that means is that we have to connect via VPN (which is our bypass software). I have the VPN software installed on every computer at the office. We load the VPN, and then log on. VPN is not meant to be used in this way, so we sometimes get kicked off the EMR, but at least it works somewhat.

I have a list of instructions posted on the wall besides my router, just in case this happens when I am not there. I have shown my practice partners and my office administrator, so they know what to do. There is no help or manual to figure this out, and the way I have done it is by learning from my mistakes with each subsequent failure; I do not think that this is a good way of planning for problems.

When the eHO lines go back up, we reverse the process. We have to then wait for 5 minutes for the eHO internet lines to connect, so I either wait for a call from our IT guy letting us know that we are good to go, or we try at lunch. We can't do this in the middle of the office, as physicians are seeing patients, and staff are fielding calls and incoming patients at the front.

Today, three HydroOne transformers blew near my hospital, so all internet connections coming into the hospital are affected. We have no service at all, even through backup. We are now starting to wonder if we should have a backup line going to the server--if eHO allows this.

These issues have affected our decision on ASP. The ASP server is hosted at eHO's facility. Their service and communication (we still don't know what happened Tuesday) have been consistently less than stellar--and we are still off-line at the time I am writing this. It would be fair to say that we do not trust that organization with hosting a server managing our data; I think they will have to demonstrate improvements in reliability and communication before we reconsider.

We have to deal with eHO for the lines coming to our new office. I have just received forms for the Order Agreement, which I do not fully understand. I will need help filling those out; I am now frankly worried about whether that organization can supply the connectivity for the 80 to 100 people who will be accessing the EMR remotely in the new location. I guess we'll have to see what happens.


1 comment:

Dr. David M. Kaplan said...

Michelle. EHO now also you to connect directly to your EMR ASP (i.e. Nightingale on the Ontario Server) via a Rogers or Sympatico line. No VPN needed anymore. This is make having a 'backup' connection much more straightforward for ASP users.