“The odd call drops” of the Mediation Server

I had a very annoying issue lately when an installation of a new gateway resolved in some calls (specifically to US numbers) dropped by the Skype for Business mediation server saying “A call to a PSTN number failed due to non availability of gateways.

The cause, according to the mediation server, was that “All gateways available for this call are marked as down“, and the resolution, surprisingly, was to “Verify that these gateways are up and can respond to calls.”

It seemed funny, because all other calls were successful, I have not exhausted the available PRI channels I had, the gateway didn’t seem to lose connectivity for a split second and SIP options are accepted and replied to on both ends.

Looking further at the Lync Monitoring Reports, I noticed the following:

Reported by Client
12000; reason=”Routes available for this request but no available gateway at this point”

Reported by Server
12000; reason=”Routes available for this request but no available gateway at this point”

I went to the gateway (AudioCodes M1000B) for answers and the logs were showing something very weird;

Every call starts with the same flow:

Flow1The mediation server sends a SIP Invite that will normally be answered immediately by the gateway with a “100 Trying”. This, for me, will close the deal on the “Gateway no responding” – this is a great response if you ask me.
The next phase would be the gateway immediately sending a “PSTN Place call” to the PSTN, hopefully receiving a “Proceeding” instantly from the PSTN, meaning so far we’re fully communicating. So for the time being I’m not really buying what the Lync mediation server is selling.

Looking further at the logs, I see the following strange behaviour:
As I’m still waiting for the PSTN to connect the call (this would normally be a “PSTN Call Alerting” message), I see that the mediation server decided to abandon ship!
This is what I see in the logs:

Wrong

Within 10 seconds after initiating the call, the mediation server will send a “Cancel” request to the gateway and will terminate the call:

19:25:49.640 [S=660986] [SID:1327438938] INVITE sip:+353857560598@10.10.10.1;user=phone SIP/2.0 
19:25:49.729 [S=661027] [SID:1327438938] SIP/2.0 100 Trying 
19:25:49.730 [S=661035] [SID:1327438938] (   lgr_psbrdif)(    656225)   pstn send --> PlaceCall
19:25:50.087 [S=661039] [SID:1327438479] (   lgr_psbrdex)(    656229)   pstn recv <-- CALL_PROCEEDING
19:25:59.020 [S=661055] [SID:1327438938] CANCEL sip:+353857560598@10.10.10.1;user=phone SIP/2.0 
19:25:59.052 [S=661061] [SID:1327438938] SIP/2.0 200 OK 
19:25:59.058 [S=661065] [SID:1327438938] SIP/2.0 487 Request Terminated 
19:25:59.079 [S=661074] [SID:1327438938] (   lgr_psbrdif)(    656265)   pstn send --> PSTNDisconnectCall

Clearly, the gateway is available and responding, but the mediation server is somewhat impatient.

Doing a bit of digging around, it appears there’s a setting for this.

The Lync mediation server is expecting a “SIP 180 Ringing” within 10 seconds of initiating the call. Failing to provide that (183 “Session in Progress” can be pushed here but that won’t benefit you if you’re still waiting for that ring…) will cause the mediation server to decide that the call didn’t respond and to mark the gateway as unavailable. A little too harsh? In my opinion yes. This will generate errors (2 per failed call) on your Lync servers and is actually a Lync self-inflicted error.

To work around this issue you can go to:
<Lync or Skype for Business installation folder\Server\Core
and look for a file called “OutboundRouting.exe.config“.
When you open that file, the first configuration line is:
<add key=”FailOverTimeout” value=”10000″/>
This actually determines how long should a call wait for the gateway until it declares it “Offline”. the value is in milliseconds, meaning the default setting is 10 seconds.

This will impact your environment if you have two or more gateways, and it means that it’ll take the mediation servers more than 10 seconds to declare that gateway as offline if it actually fails.

For me to workaround that issue I extended the wait time to 20 seconds, restarted the front-end and mediation services, and all of a sudden no call was dropped.

It appears it took the carrier approx. 12 seconds to generate the “PSTN Call Alerting” command to the gateway. For some reason, only calls to the US (and only with this specific carrier) were affected by this issue.

Now, I can see all my calls working perfectly and the errors have disappeared from the serves too.

Advertisements

Skype for Business VCFG Viewer

When preparing documentation for a customer or trying to survey an existing deployment, I usually ask for a copy of the Voice Configuration files (.vcfg) of Lync \ Skype for Business to see how things are set up.

Currently there’s no option to view the vcfg file in a readable format.
Sure, you can open the vcfg file in a text editor or even a web browser and browse through the xml. Alternatively, you can import the configuration to your servers – but good luck trying to read it between the “Committed” and “Uncommitted” text.

The vcfg viewer is a “PowerShell executable” that runs through the vcfg file and renders it to a readable html file.
Simply run the ps1 file, choose the vcfg file you wish to open, and you’ll have a browse able page to examine the vcfg file:

Download the ps1 file here.

Lync 2013 E.164 CLI and carriers (It’s that Plus sign!)

If you’re using Lync properly, you must have all your numbers in accordance with the E.164 format, as recommended by Microsoft and as often preached by Ken Lasko.

There are many benefits for using E.164, but – like any good thing – it comes with some flaws. One of these flaws when using Lync 2013 is that Lync will cease normalizing numbers that are prefixed with +.
The moment you add the ‘+’ sign to your number, Lync will consider it already normalized and will not process any more rules.

This can cause issues with some providers that require us to remove the leading ‘+’ sign from the number.
The work around for dialled numbers (Or as Lync describes it “Called Numbers”) is to use the Set-CsTrunkConfiguration command to remove ‘+’ sign at the trunk level by adding the parameter -RemovePlusFromUri $true.

This is all good for numbers Called – but what about numbers Calling? How can we manipulate the “Source” number?
If we look at a trace from Lync we can see that the destination number is stripped from it’s leading ‘+’, but the source number still has that annoying sign, although I have a rule that says it’s removing it:

With plus

The SIP invite shows the issue:
INVITE sip:3531891170170@MediaGateway.y0av.local;user=phone SIP/2.0
FROM: <sip:+35314396804;ext=6804@y0av.com;user=phone>;epid=3DFA3756A7;tag=1e448bc93
TO: <sip:3531891170170@MediaGateway.y0av.local;user=phone>

As you can see in the image above, the ‘+’ sign is removed from the destination number, but it’s still there on my source number.
This might result on certain carriers dropping the call since they’re expecting the CLI with no ‘+’.

The workaround:

If you’re working with a gateway or a SBC, you’ll usually be able to work around this issue by stripping the ‘+’ there, simple as this.

If you’re working with Lync only and using a direct SIP trunk to your provider – it’s easy too!
The numbers are not normalized due to Lync’s lack of ability to ignore the rest of the data in the FROM line; when we’re creating normalization rules within Lync, we will usually be looking at numbers and specific characters, not the entire weird string.
The workaround then, is to treat the entire string as… a string.
I add the following rule for my “Calling Numbers” manipulation in the trunk:

Rule

This rule says that whenever a number starts with a plus and has at least 8 characters (ANY character) following it, we should ignore the ‘+’ and send only the characters:
Pattern to match: ^\+(.{8,})
Translation rule: $1
Of course, you can change that to look for more numbers if you have other rules there.

Once the rule has been placed and committed, traces look like this now:

Without Plus

The SIP Invite shows the difference:
INVITE sip:3531891170170@MediaGateway.y0av.local;user=phone SIP/2.0
FROM: “Yoav Barzilay”<sip:35315267877;ext=7877@y0av.com;user=phone>;epid=3DFA3756A7;tag=c9b24cf3b
TO: <sip:3531891170170@MediaGateway.y0av.local;user=phone>