“The odd call drops” of the Mediation Server

I had a very annoying issue lately when an installation of a new gateway resolved in some calls (specifically to US numbers) dropped by the Skype for Business mediation server saying “A call to a PSTN number failed due to non availability of gateways.

The cause, according to the mediation server, was that “All gateways available for this call are marked as down“, and the resolution, surprisingly, was to “Verify that these gateways are up and can respond to calls.”

It seemed funny, because all other calls were successful, I have not exhausted the available PRI channels I had, the gateway didn’t seem to lose connectivity for a split second and SIP options are accepted and replied to on both ends.

Looking further at the Lync Monitoring Reports, I noticed the following:

Reported by Client
12000; reason=”Routes available for this request but no available gateway at this point”

Reported by Server
12000; reason=”Routes available for this request but no available gateway at this point”

I went to the gateway (AudioCodes M1000B) for answers and the logs were showing something very weird;

Every call starts with the same flow:

Flow1The mediation server sends a SIP Invite that will normally be answered immediately by the gateway with a “100 Trying”. This, for me, will close the deal on the “Gateway no responding” – this is a great response if you ask me.
The next phase would be the gateway immediately sending a “PSTN Place call” to the PSTN, hopefully receiving a “Proceeding” instantly from the PSTN, meaning so far we’re fully communicating. So for the time being I’m not really buying what the Lync mediation server is selling.

Looking further at the logs, I see the following strange behaviour:
As I’m still waiting for the PSTN to connect the call (this would normally be a “PSTN Call Alerting” message), I see that the mediation server decided to abandon ship!
This is what I see in the logs:

Wrong

Within 10 seconds after initiating the call, the mediation server will send a “Cancel” request to the gateway and will terminate the call:

19:25:49.640 [S=660986] [SID:1327438938] INVITE sip:+353857560598@10.10.10.1;user=phone SIP/2.0 
19:25:49.729 [S=661027] [SID:1327438938] SIP/2.0 100 Trying 
19:25:49.730 [S=661035] [SID:1327438938] (   lgr_psbrdif)(    656225)   pstn send --> PlaceCall
19:25:50.087 [S=661039] [SID:1327438479] (   lgr_psbrdex)(    656229)   pstn recv <-- CALL_PROCEEDING
19:25:59.020 [S=661055] [SID:1327438938] CANCEL sip:+353857560598@10.10.10.1;user=phone SIP/2.0 
19:25:59.052 [S=661061] [SID:1327438938] SIP/2.0 200 OK 
19:25:59.058 [S=661065] [SID:1327438938] SIP/2.0 487 Request Terminated 
19:25:59.079 [S=661074] [SID:1327438938] (   lgr_psbrdif)(    656265)   pstn send --> PSTNDisconnectCall

Clearly, the gateway is available and responding, but the mediation server is somewhat impatient.

Doing a bit of digging around, it appears there’s a setting for this.

The Lync mediation server is expecting a “SIP 180 Ringing” within 10 seconds of initiating the call. Failing to provide that (183 “Session in Progress” can be pushed here but that won’t benefit you if you’re still waiting for that ring…) will cause the mediation server to decide that the call didn’t respond and to mark the gateway as unavailable. A little too harsh? In my opinion yes. This will generate errors (2 per failed call) on your Lync servers and is actually a Lync self-inflicted error.

To work around this issue you can go to:
<Lync or Skype for Business installation folder\Server\Core
and look for a file called “OutboundRouting.exe.config“.
When you open that file, the first configuration line is:
<add key=”FailOverTimeout” value=”10000″/>
This actually determines how long should a call wait for the gateway until it declares it “Offline”. the value is in milliseconds, meaning the default setting is 10 seconds.

This will impact your environment if you have two or more gateways, and it means that it’ll take the mediation servers more than 10 seconds to declare that gateway as offline if it actually fails.

For me to workaround that issue I extended the wait time to 20 seconds, restarted the front-end and mediation services, and all of a sudden no call was dropped.

It appears it took the carrier approx. 12 seconds to generate the “PSTN Call Alerting” command to the gateway. For some reason, only calls to the US (and only with this specific carrier) were affected by this issue.

Now, I can see all my calls working perfectly and the errors have disappeared from the serves too.

Advertisements

Irish Public Holidays 2016 set for Skype for Business Server

New year, new holiday set.

This updated script will add all 2016 public holidays (and 2017 new years day) to your Skype for Business Server \ Lync 2013 Server environment as a holiday set.

Holidays in this script:

  • St. Patrick’s day.
  • Easter Monday.
  • Bank holidays for May, June, August, October.
  • Christmas day.
  • St. Stephen’s Day.
  • New years day 2017.

This will then become an option to tick on your response group workflows:

2016

The script can be downloaded here.

Skype for Business VCFG Viewer

When preparing documentation for a customer or trying to survey an existing deployment, I usually ask for a copy of the Voice Configuration files (.vcfg) of Lync \ Skype for Business to see how things are set up.

Currently there’s no option to view the vcfg file in a readable format.
Sure, you can open the vcfg file in a text editor or even a web browser and browse through the xml. Alternatively, you can import the configuration to your servers – but good luck trying to read it between the “Committed” and “Uncommitted” text.

The vcfg viewer is a “PowerShell executable” that runs through the vcfg file and renders it to a readable html file.
Simply run the ps1 file, choose the vcfg file you wish to open, and you’ll have a browse able page to examine the vcfg file:

Download the ps1 file here.

Skype for Business Response Groups Diagrams

This is something I was looking for every time I had a deployment with multiple Response Groups.

Customers and management would usually like to see a graphic chart of how the Response Groups’ workflows work, what are the timeout and over flow options, what’s the chosen routing method for each group,  and who are the users in each group.

Looking at Matt Landis‘ great Get-csRgsWorkflowDiagram script, I was inspired (and in need) to do something similar.

Took a while to get it where I wanted, but the version published here now is one I’m very happy about and it actually gives me everything I was looking for.

It’s actually two files;

  • One file is the script that you should run from your PC.
  • The second file is a custom Visio stencil (.vss file) that will be downloaded to your “My Shapes” folder (That’s always in your “My Documents” folder after you install Visio) and will provide the Visio images for the script. You can find it here if you wish to download it to your machine beforehand.
    The script will detect proxy\download issues and will prompt you to download the VSS file manually if required.

Prerequisites

  • Run this script from your workstation – not from the server. It doesn’t matter if your workstation is in the domain or not.
  • You’ll need an installation of Visio on that machine.
  • PowerShell 3.0 is a must.
  • ADDS RSAT is required if you’re sing Active Directory Distribution Groups for some of your Response Groups’ groups. We’re using the AD PowerShell Module to expand these groups and list their users.
  • Your Lync\S4B pool’s FQDN and administrator credentials.
  • Your Execution policy set to Bypass, at least until I start signing my scripts…

Running the script

From PowerShell, run the file, it will immediately prompt you for your Pool’s FQDN:

Pool Name

Enter your pool’s name and you’ll be prompted for credentials:

Creds

The script will generate a list of all your Response Groups. Pick up a Response Group nunmber from the list, or hit 0 for all Response Groups. Each workflow will be drawn in a new page:

WF List

If required, it will download the Visio stencil and place it in your “My Shapes” folder:

VSS

The script will start drawing the diagram in the background:

Drawings

Upon completion, the script will auto-save the file to your My Documents folder:

Path

The file is saved with the workflow’s name and the creation date:

File

And is ready to be opened:

Visio Window

For multiple Response Groups you’ll have multiple pages, each with the Workflow’s name.

Diagrams are scaled to fit the pages, but might be larger due to large number of agents in a group.

Known Issues

  • This script was tested in English, you might experience some issues if you’re running Visio under a different language.
  • Windows 10 OS build 10565 can’t install ADDS RSAT – the script will not expand Distribution Groups.

Download

Please download the script here.

Reset \ Fix Skype for Business and Lync accounts …And a bonus!

Every now and then I run into one account that has something weird about it. Usually it would display an old phone number even when you changed the phone number in AD, ran Update-CsAddressBook and deleted all old records you had.
However, when looking at the user’s contact card you’ll still see the old number AND the new number, resulting in users dialling the wrong number, resulting in call failures, resulting in service desk calls, do I need to go further…?

I read various workaround involving running various scripts against the RTC database, and to be honest – there are some thing I’d rather not touch.

Workaround for the “corrupted” users would normally be disabling them and then re-enabling the for Skype for Business \ Lync.
This will do the trick, BUT has a huge downside: if you didn’t export the users’ data they’ll lose all of their groups, favourites, contacts, etc. Also, you’ll have to re-apply the user’s policies, line URI, private numbers, etc.
This is both time consuming and would require to schedule a maintenance window as the user is going to be kicked out.

So, came up with the following script:

Window

  1. You’ll provide the user’s sip address.
  2. The system will confirm the user’s name so you’re happy to move on.
  3. The user’s data will be exported to a folder on your local C drive (Path is always C:\y0av\users\<username>)
  4. The user’s Get-CsUser data will be exported to a text file in the same folder so you can compare the settings once you’re done.
  5. The user will be disabled, and the script will run Update-CsUserDatabase and pause for 15 seconds.
  6. The user will then be enabled and the script will pause for another 15 seconds. The purpose of the pauses is to allow for the changes to set in. I tested various environments and 30 seconds seems like enough time. I’ll add the option to change the pause in future versions.
  7. The user’s polices will be re-applied.
  8. The system will run Update-CsUserDatabase and Update-CsAddressBook to reflect the changes.
  9. The user’s data dump is saved in the C:\y0av\users folder, and you can manually delete it if you feel you no longer need it.

Folder

What’s missing?

  • If the user is configured with a Private Line you’ll have to use the script with the -PrivateLine switch.
  • PIN must be reset for the user after the account is recreated.
  • Conference ID will change – any recurring Skype for Business \ Lync meetings must be re-sent.

What’s the bonus?

All the tests I made show that the script runs fast enough to not interfere with the user’s activity; I ran this against a user during a call, the script completed and the call never disconnected.
Try this on a test account before killing someone’s call in your organization… 🙂

______________________________________________________________________________

Policies this script will save and re-apply:

Hosted Voicemail (True or False)
Archiving Policy
CallViaWork Policy
Client Policy
ClientVersion Policy
Conferencing Policy
External Access Policy
Hosted Voicemail Policy
Location Policy
Mobility Policy
Persistent Chat Policy
Pin Policy
Presence Policy
Third Party Video System Policy
User Services Policy
Voice Policy
Voice Routing Policy

To run the script for a user with a private line, run:

S4BUserRepair.ps1 -PrivateLine

Download the script here.

Skype for Business Server – Assign User policies to AD groups

This is an update to a previous version of this tool written by myself and Guy Bachar.
The updated version can now run (and was tested!) on both Lync server 2010 and 2013, and Skype for Business Server.

Run this tool from Lync Management Shell or Skype for Business Management Shell.

Note you will need to run this tool with Local Admin permissions (You will be prompted for elevation automatically if not) and you must have ADDS RSAT installed so you can use the Active Directory PowerShell Module.
You will be asked to provide an Active Directory Group name. Type the Display Name of the group, the tool will reply with the CN of the group, confirming you chose the right group:

Choose Group

Then choose the type of policy (or dial plan) you want to assign this group, there are 14 options:
1     Voice Policy
2     Client Policy
3     External Access Policy
4     Mobility Policy
5     Archiving Policy
6     Hosted Voicemail Policy
7     Client Version Policy
8     Conferencing Policy
9     Voice Routing Policy
10     Location Policy
11     PIN Policy
12     Presence Policy
13     Persistent Chat Policy
14     Dial Plan

After choosing one of the 14 options, you’ll be asked whether you want to assign the global policy to this group or choose from the existing user policies:

Choose from policies

If you choose 1 (This is always the Global policy), that will be assigned to the group.
If you choose 2, a new sub-menu will open, detailing the policies you can assign:

External Access

You will then be asked to confirm the change and the policy will be assigned the group.

The tool can be downloaded here.

 

 

 

‘Tis the season to prepare – Add Ireland Public holidays for 2015 to Lync Server

It’s this time of the year again, and the most common question is “What do we do when we’re going on holidays?”.

The answer is here! (That is, if you live in Ireland :)) – the attached script will add all Public holidays for 2015 …And then some:

According to the Citizen Information Board, the following are Public Holidays in Ireland in 2015:

  • New Year’s Day (1 January)
  • St. Patrick’s Day (17 March)
  • Easter Monday (This year it’s 4 April)
  • First Monday in May, June, August (4 May, 1 June, 6 August)
  • Last Monday in October (26 October)
  • Christmas Day (25 December)
  • St. Stephen’s Day (26 December)
    Good Friday is not a public holiday.

The script will add all of the above and the following:

  • Christmas Day 2014 (25 December)
  • St. Stephen’s Day 2014 (26 December)
  • New Year’s Day 2016 (1 January)

You will need to check the new holiday set box for your RSG Workflows to activate your holiday configuration.

To run the script, simply run it one of your Lync servers. The script will confirm that you have local admin permissions and will prompt you for automatic elevation if necessary.

then, it will search for and display your existing application servers, please choose the one(s) running your RGS.
The new improvement to the script is that it will ask you whether you’re using a 12-hour or 24-hour configuration on your machine to determine how to run.
Once you choose 12 or 24 hour configuration, the script will run and add a holiday set called “Ireland Public Holidays 2015”.

Holidays

Download the script here.

List Lync Users – automated Excel file

(Have a look at the newer, smarter, prettier, fancier version of this script here.)

This script isn’t big news, but it’s very handy and easy to use.
It allows you to remotely connect to one of your pools and create an Excel sheet, populated with the following:

Display Name SIP Address Registrar Pool Enterprise Voice Line URI

It’s basically a list you can hand to your customers and colleagues so they can keep track of changes.
Additionally, it helps spot Enterprise Voice enabled users that don’t have a Line URI or it’s misconfigured.

When you run the script it’ll ask your for your Credentials – You must Have appropriate Lync permissions:

Credentials

It will then ask you for the FQDN of your Lync server or pool and will attempt to connect:

Lync Server

When connected, the script will immediately start creating the Excel file:

Creating Excel2

You can scroll up and down the Excel file but do not click anywhere on the spreadsheet. This causes the automated build to crash, and you’ll have to restart the script.

When completed, the script will notify you and close the remote PowerShell session it initiated.

Script Done

The pace is approx. a second per line, so if you have 5000 users this is the time to go and grab a cup of coffee.

The script can be downloaded here.

Updated October 7, 2014: Version 2.0 checks for your PS version. This script will only run on PS 3.0 or higher.