Configuring Inbound Hybrid Search for a Large Distributed Organization

Configuring Inbound Hybrid Search for a Large Distributed Organization

In this post I'm focusing on a particular problem area to be aware of when configuring the Inbound Hybrid Search scenario. I'd like to call out some of my personal experiences when working through this scenario with a real organization, supporting a large number of users across a large number of devices.

There are many great articles out there around configuring Hybrid Search scenarios for SharePoint, particularly for the Outbound Hybrid Search scenario, and somewhat for the Inbound Hybrid Search scenario. To clarify the difference between these scenarios, Inbound Hybrid Search is about surfacing search results for content from your SharePoint on-premises environment in your Office 365 SharePoint Online tenant. The Outbound Hybrid Search scenario on the other hand, is pretty much the opposite, i.e. surfacing content from your SharePoint Online tenant in your SharePoint on-premises environment.

I won't be repeating the well documented subject of configuring the Outbound Hybrid search scenario in this article, as Microsoft already have good documentation on the matter. In addition a friend of mine, Manas Biswas, has covered the subject extremely well in his own multi-part blog series.

Problem Scenario

Imagine a scenario where you want to configure the Inbound Hybrid Search scenario as described in the aforementioned articles, i.e. you have a Search Centre in your SharePoint Online tenant, for which you want to provide an enterprise search capability to your users, so that as well as surfacing content from the local SharePoint Online sites, you can also surface content from your on-premises environment.

Without dwelling too much on the OAuth authorization flow right now, this scenario relies on a trust relationship between SharePoint 2013 and Office 365. We rely on Client Certificate Authentication to authenticate requests made by SharePoint Online in the SharePoint on-premises environment. This can pose certain issues for employees working remotely in the SharePoint Online tenant, or coming from outside the domain for any other reason.

In an Inbound Hybrid Search topology, SharePoint Online queries the on-premises SharePoint farm using a public URL, for example https://team.hassanionprem.com. This URL resolves to an endpoint on a reverse proxy that's configured to first pre-authenticate requests from SharePoint Online with a client authentication certificate, and then relays the request to the SharePoint farm.

After the query reaches the on-premises SharePoint farm and is processed, search results are sent back to SharePoint Online. SharePoint Online renders content to remote users by using the web application's public URL. This includes rendering search results' URLs, for example https://team.hassanionprem.com.

But when the remote user logs in and clicks an on-premises document link in the search results (https://team.hassanionprem.com/Documents/document.docx), the reverse proxy responds by requesting the client certificate from the user's computer. If this certificate is not present, the reverse proxy cannot pre-authenticate the request, and returns an unfriendly error message. Certificate deployment can be managed, but this problem is further exasperated when you consider a large user base, working in a multitude of locations across differing devices.

The Solution

The May 2014 Cumulative Update for SharePoint 2013 provides a solution to this dilemma. The update provides a fix that allows you to enable a fix for Alternate Access Mapping (AAM) matching to overcome the issue.

The fix is described in this Microsoft article, however is not entirely accurate, or at least not particularly clear.

Nevertheless, the change introduced overcomes an issue prevalent prior to applying the fix which allows you to add a second reverse proxy endpoint to be used by users accessing content through search results.

Ultimately, in the simplest of scenarios you would end up with configuration as described in the table below:

 

Published Endpoint Published URL Internal URL Authentication Purpose
​1

https://teamspo./

hassanionprem.com

https://teamspo.

hassanionprem.com

Client Certificate Serves SharePoint Online requests
​2

https://team.

hassanionprem.com

https://team.

hassanionprem.com

Windows Serves User's requests for accessing content

 

Published Endpoint 1 would be used to process requests from SharePoint Online as part of the OAuth authorization process to retrieve security trimmed search results from your on-premises farm, versus requests from users accessing on-premises SharePoint content against Published Endpoint 2.

Published Endpoint 2 can pre-authenticate user requests for on-premises SharePoint content by using Active Directory Federation Services (ADFS), Forms Based Authentication (FBA), or any other authentication methods that are available.

In my examples below I actually use Web Application Proxy (WAP) and use pass-through authentication rather than taking advantage of any pre-authentication capability (which isn't necessarily recommended for production scenarios - see my previous post on publishing SharePoint via WAP).

Implementation

Install the patch and enable the fix:

To go ahead and get this up and running, the first thing you'll need to do is deploy the May 2014 Cumulative Update (CU) or a more recent CU (they are cumulative by definition). Once it has been deployed, you can then run the following PowerShell to enable the fix for AAM matching to work as intended:

$config = Get-SPSecurityTokenServiceConfig

$config.UseIncomingUriToValidateAudience = $true

$config.Update()

Configure AAMs in SharePoint:

To configure AAMs in my simple scenario outlined in the table above, I took my web application and added an internal URL for the URL that will be used by SharePoint Online queries. Note that in my example users access content on the same URL internally as that which is to be made publically available.

Alternate Access Mappings in SharePoint

From here you then need to manually add a binding in IIS on each SharePoint server in your farm for the URL that will be used by SharePoint Online, in my case https://teamspo.hassanionprem.com, and ensure you also bind the correct certificate.

You may be thinking that typically you would extend a web application to a new zone in this sort of scenario, however the AAM matching relies on the URLs to be in the same zone.

Publish Endpoints

The final piece of configuration is to publish your two endpoints in your chosen Reverse Proxy device. In my example I use WAP, but there are other supported options for SharePoint Hybrid scenarios that have been documented by Microsoft.

Publishing the endpoint to server user content is no different to publishing in normal scenarios, I've discussed options with WAP in previous blog posts using pass-through or pre-authentication options.

Publishing the endpoint that will serve SharePoint Online requests must use the Client Certificate authentication option, and can be achieved running PowerShell as per the below:

Add-WebApplicationProxyApplication -ExternalPreauthentication ClientCertificate -ExternalUrl https://teamspo.hassanionprem.com -BackendServerUrl https://teamspo.hassanionprem.com -name "Team SPO" -ExternalCertificateThumbprint -ClientCertificatePreauthenticationThumbprint -DisableTranslateUrlInRequestHeaders:$False -DisableTranslateUrlInResponseHeaders:$False

Note that the Certificate referenced in the above PowerShell must already be installed on the WAP server, and will be stored in the Secure Store in your SharePoint Online Tenancy (as outlined in Manas' aforementioned blog post). One additional recommendation is that the certificate you use for this purpose isn't the same certificate that you use to publish your endpoint that serves user content (https://team.hassanionprem.com).

That's it! … Well almost…

You're now pretty much in a position where you can go ahead and configure the Inbound Hybrid Search scenario pre-requisites and workload specific configuration all outlined in Manas' blog post.

If you've gone ahead and installed the May 2014 CU as suggested above to configure this Hybrid scenario, you will have also by default have installed fixes included in the April 2014 CU. Unfortunately the April 2014 CU includes a regression which breaks the Inbound Hybrid scenario.

When you get to the point of configuring the Inbound Search piece, more specifically you attempt to configure the query rule to display results from your remote index (SharePoint on-premises) result source, you are presented with an unauthorized error rather than the desired results set.

The error will read something like this within the query builder window:

System.Net.WebException: The remote server .returned an error: (401) Unauthorized. at System.Net.HttpWebRequest.GetResponse() at Microsoft.SharePoint.Client.SPWebRequestExecutor.Execute() at Microsoft.SharePoint.Client.ClientContext.GetFormDigestInfoPrivate() at Microsoft.SharePoint.Client.ClientContext.EnsureFormDigest() at Microsoft.SharePoint.Client.ClientContext.ExecuteQuery() at Microsoft.Office.Server.Search.RemoteSharepoint.RemoteSharepointEvaluator.RemoteSharepointProducer.RetrieveDataFromRemoteServer(Object unused) at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) at Microsoft.Office.Server.Search.RemoteSharepoint.RemoteSharepointEvaluator.RemoteSharepointProducer.ProcessRecordCore(IRecord record)

Unfortunately at the time of writing this article, there is no fix, however, there is a workaround available, which involves running the following PowerShell:

$config = Get-SPSecurityTokenServiceConfig $config.AuthenticationPipelineClaimMappingRules.AddIdentityProviderNameMappingRule("OrgId Rule", [Microsoft.SharePoint.Administration.Claims.SPIdentityProviderTypes]::Forms, "membership", "urn:federation:microsoftonline") $config.Update()

The above issue is described further in this Microsoft support article.

Now you're ready!

You're good to go, assuming you've followed the above points and the general guidance provided by Microsoft TechNet or Manas, you should be well on your way to presenting search results from your on-premises SharePoint farm in a Search Centre in your SharePoint Online tenant. In the below screenshot you can see on-premises results being served from https://team.hassanionprem.com and local SharePoint Online results at https://brightstarrdevelopment.sharepoint.com.

Sam Hassani Principal Consultant

‘Sensible’ Sam is an asset to the consulting team. Having jumped the fence from Microsoft a few years back, he now consults and designs large scale SharePoint, O365 and Azure solutions for BrightStarr. His conscientious approach got him his nickname, and although he’s not afraid to take risks, he’s always a safe bet. Delivering stable, well thought out solutions time and time again.

In his previous job, he specialized in SharePoint, working in the field and with the product team, Sam is genuinely excited by his vocation and speaks at conferences and events with enthusiasm. Just don’t get him started on hybrid search! At the weekend Sam likes to ride his road bike and hang out with his young son.

Related Tags
Twitter Feed

Where Next? Relevant Stories and Insights.

What is a Hybrid SharePoint System?
What is a Hybrid SharePoint System?
A hybrid SharePoint system is one that uses both SharePoint ‘On Premises’ and SharePoint Online (part of Office 365) to deliver an integrated solution. It in effect combines a private traditional SharePoint system with a publicly hosted cloud instance. For some clients, and projects (for example hybrid intranets), it offers the best of both worlds.
The Azure Search Service
The Azure Search Service
The Azure team has been steadily adding features to their Pay-as-you-Go Search Service. At the time of writing it was still in preview mode, but the feature set is now starting to represent a respectable cloud Search Service.
The Cloud Search Service Application has been released for SharePoint On-premises
The Cloud Search Service Application has been released for SharePoint On-premises
Not so long ago I spoke about the new Hybrid Search experiences provided by the all new Cloud Search Service Application (SSA) that was under development by Microsoft and beta tested by a selection of partners including BrightStarr in this article.