Tuesday 19 April 2011

VXML server isolation

Our IVR VXML Seaside server needs to handle up to 700 calls per hour. Normally that's not a problem, but we do get a few timout errors per week, which is why we're moving to a three image load balanced deployment.

Recently we had another issue with our current setup: the server hosting the file share, to which call files are written, failed. Calls were still processed, but the final write failed. Write are done in a forked block with an exception handler, so the image dealt with the errors cleanly. And the call data was all written to a log, so no data was lost.

However, there was a performance hit on the server image and we started to get timeout errors on more than 10% of the calls. So we turned off the Seaside server which triggered an automatic failover to our AWS hosted server. It works, but it's a bit slow. Callers could wait several seconds for the call prompts (we'll be upgrading it soon).

This was the first time we had an error with the file share and it pointed out a dependency that we'd prefer to avoid. Ideally, the VXML servers would continue to process calls even if other servers are down; they should as isolated as possible.

Our strategy is to use the VW Opentalk image to image interface and have a data server gather call information from the three VXML servers. It will also send them updated validation files. This way, the VXML servers have no folder share dependencies,  the data server can show status across all three VXML server, and the VXML servers can function even if the entire back office is down. And we can host the data & VXML servers with different service providers.

We've been using Opentalk in our client patch system for a couple of years now. It's familiar and reliable. And ya gotta love the deployment flexibility you get with Smalltalk.

No comments:

Post a Comment