When a Customer's Network Service Outage is a "Good Thing"  
July 18, 2008

by Dan  Baldwin, TA Founder

For the actual network service customer there can be no "good" time to have your voice, data or Internet services stop working - for a call center especially. Employees are idled. Clients are aggravated. Owners loose money. In rare occasions though, a telecom agent can benefit from an outage - especially if the agent (or the carriers the agent has recommended) can't be blamed for the outage. The reason an outage can be good is that the outage incident gives the agent and the customer a unique opportunity to measure the carrier's response to the outage.

A particular case in point is an outage that occurred to one of my call center customers last month. It started out as the standard agent nightmare when your cell phone won't stop ringing at 9am on a Saturday morning while you're trying to sleep in. I know I was in trouble when I saw the caller ID of my customer's IT manager on my cell phone. In short order I learned that all eight of his voice T1s were down hard - no dial tone, no lights on the equipment. While I was sad for the customer I was happy for myself as a year ago I had shrewdly recommended that the customer split the eight voice T1s between two different resellers "to help keep the resellers honest".

Even though both resellers had my customer on the same underlying carrier (Qwest in this case) I was fairly sure that Qwest was not the source of the problem because I has another call center across town also on Qwest. If Qwest was the problem I would have two upset customers calling me on a Saturday morning, not just one. I told my customer that the problem was most likely with the local loop provider, AT&T and that I'd have a trouble ticket started with both resellers within minutes.

When I suggested AT&T was the problem I set myself up with my customer. From the moment "AT&T" came out of my mouth, the only proper resolution to the problem from my customer's point of view was an AT&T repair truck parked in front of his business. Unfortunately, AT&T was "two vendors away". Not only could I not call AT&T, I couldn't even call Qwest as neither vendor recognized my customer as their customer. But this is where the "good part" started as I figured I'd call both resellers I had sold the customer to see which of them would get Quest to get an AT&T truck dispatched quickest. One reseller would win and one reseller would lose the "customer outage contest" this weekend. After making the next two calls it seemed like the contest was already over.

Reseller "A" excelled from the start. I didn't have either reseller's escalation list with me so I went to the resellers' web site. For reseller "A" I simply clicked on the "support" link on their website and there it was, all the escalation phone numbers. I called the first number on the list and was pleased to have reseller A's weekend repair tech answer the phone. He could not have been more helpful. He took the customer's name, all the trouble shooting information I had collected and promised to have Qwest involved ASAP.

Reseller "B" did not excel from the start. Their escalation list was not on their website so I called the posted customer service number only to be connected (after three minutes on hold) with reseller B's answering service. I told the answering service that I had a call center customer out of service and requested that they page the on-call technician to call me ASAP. I did get a call from the reseller's weekend repair tech within 10 minutes but was somewhat horrified with the young man's request. "You'll have to call the answering service back and give them all the circuit ID numbers that are down and some sample calls that are not going through. We can't get a trouble ticket started until we get all that information."

I thought I had forgotten all the really bad cuss words I learned in the Navy over 20 years ago. I had not. After exhausting my salty list of profanities on the fellow I explained as succinctly as possible the he needed to open his laptop computer, log into his own customer database, get the circuit IDs himself and call Qwest. He of course assured me that's not the way their process works. I assured him that if he couldn't get the trouble ticket started that I'd place the trouble ticket with the president of the company within the next five minutes. He grumbled. I hung up and called the his supervisor, his department head and the president - I finally found the escalation list on the reseller's agent web site.

To make a long story short, calling the second reseller's president worked great because within 30 minutes bother resellers were working equally hard to grind through the Qwest and then AT&T trouble shooting process to get the trouble fixed. A couple hours later it was reported that a car had run into an AT&T wire facility and that several business parks in the area were all without service. I told the customer that if they wanted to have any hope of salvaging their weekend of dialing that he might send one of his managers out to see if he could find the AT&T truck working on repair job and beg to have his building put back in service first. The customer did that and they got service restored by midnight and were able to salvage their Sunday calling.

Lessons Learned from an Outage

1. Be brutally honest with your customer about repair times. Customers know you didn't break it and they know you're not directly fixing it. What they want to know from you is "what's reality" as to when this might be fixed. In talking to both my resellers they said such an outage involving a wire facility might take several days to have the local loop provider fix. I passed this information on to the customer but also suggested he go find the AT&T guy working on the problem for an honest assessment.

2. Explain "Who does what and why". While all my customer wanted to see was an AT&T repair truck parked in front of his building I explained that both the resellers, Qwest and AT&T had specific trouble testing procedures to go through and these procedures were all that guaranteed that some critical problem would not be overlooked which might unnecessarily delay repairing the problem. I also explained that since we were not direct customers of Qwest and AT&T we could not officially engage them directly. I also assured my customer though that I had personally spoke with the president of the reseller and that through the reseller we had better pull with both Qwest & AT&T.

3. Don't speak poorly of the providers. It's not like your customer is going to get network service without dealing with some provider. They know providers have problems. The customers want to be reminded why you suggested one provider over another and that they have (through you, the agent) the best possible access to resources to fix the problem. When I had trouble getting reseller B started because of one confused repair tech I did not tell my customer "reseller B's all messed up". Instead I told my customer I had spoken to the president of reseller B and they were on top of getting the problem fixed.

4. Seek economical redundancy. While the eight wired PRI voice T1s were down (almost 200 voice paths) the customer was trying to pack as many uncompressed VoIP voice paths out over a 2.0 mbps internet connection he had. The obvious problem with VoIP as a backup is that uncompressed voice on a 2.0 mbps circuit (about 30 talk paths) is not going to substitute very well for 8 PRI T1s that have 23 talk paths each. The time to experiment with VoIP voice compression as a reliable backup to PRIs is far in advance of an outage. Also to be considered is the fact that carrier redundancy is not a substitute for local loop redundancy. Both sets of my customer's reseller provided T1s were down because they both came in on the same local lop provider. The only reason their Internet T1 did not go down was because it came in on a completely separate loop provider - effective but often quite expensive.

5. Call with updates even if there is no update. Your customer has their own set of humiliations when their network service is down. Don't compound their humiliation by making them call you for updates. Tell them you'll call them every 90 minutes and then call in 90 minutes. Even if you're calling to say "I just called the carrier and the situation is the same." At least your customer will feel that you are serving them still and are on the same side of the humiliation table.

Do you agree or disagree? Forward printable comments to Dan@TelecomAssociation.com