Using BCS from PowerShell with SharePoint 2013

Running data migrations with SharePoint 2013 often involves using BCS and PowerShell if you don’t have and third party tools at hand. This can be a bit tricky to set up, but once you are familiar the the BCS configuration, the PowerShell scripts are mostly straight forward.

From my experience there are a few differences when running against BCS lists vs. regular lists. I have now been able to find a good guide for this, so from my own trial and error I wanted to share a few tips.

Get the context straight: Accessing the list gives a weird error

Running this:


$Web = Get-SPWeb "http://mybcsweb/"

$Web.GetList("/Lists/MyBcsDataList")

Gives this error:

format-default : The shim execution failed unexpectedly – Proxy creation failed. Default context not found..    + CategoryInfo          : NotSpecified: (:) [format-default], BdcException   + FullyQualifiedErrorId : Microsoft.BusinessData.Infrastructure.BdcException,Microsoft.PowerShell.Commands.FormatDefaultComman

bcs-powershell-sp2013-1

The fix:


$Site = Get-SPSite "http://mybcsweb"

$ServiceContext = Get-SPServiceContext -Site $Site # Required when working with BCS
$ServiceScope = New-Object Microsoft.SharePoint.SPServiceContextScope $ServiceContext # Required when working with BCS

$Web = $Site.RootWeb
$Web.GetList("/Lists/MyBcsDataList")

A bit more wiring is required to get the context straight for BCS to work.

Make sure you are running with administrator privileges

I use PowerShell ISE when creating and running these scripts. From time to time I accidently forget to start this with Administrator privileges. This can fool you into weird errors like this one. Short minded one can try to start adding the missing assembly references, but the real reason is the missing privileges.

Running this:

$Web = Get-SPWeb "http://mybcsweb/"
$Web.GetList("/Lists/MyBcsDataList")

Gives this error:

format-default : The shim execution failed unexpectedly – Could not load file or assembly ‘System.Data.OracleClient, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089’ or one of its dependencies. Either a required impersonation level was not provided, or the provided impersonation level is invalid. (Exception from HRESULT: 0x80070542).    + CategoryInfo          : NotSpecified: (:) [format-default], BdcException  + FullyQualifiedErrorId : Microsoft.BusinessData.Infrastructure.BdcException,Microsoft.PowerShell.Commands.FormatDefaultCommand

bcs-powershell-sp2013-2

The fix:

Search for “PowerShell ISE”, right-click it and choose “Run as Administrator”.

bcs-powershell-sp2013-3

Include ViewFields in you CAML queries

Always remember to include the fields you are trying to query on in the “<Where>” statement within “<ViewFields>”. This isn’t required on regular lists, but to get any results in return, I experienced I was required to include them querying BCS lists.

Example of CAML query in PowerShell:


$Query= New-Object Microsoft.SharePoint.SPQuery 

$Query.ViewXml =
 "<View>
 <Method Name='Read List'/>
 <Query>
 <Where>
 <And>
 <Eq>
 <FieldRef Name='ProjectID' />
 <Value Type='Text'>100</Value>
 </Eq>
 <Eq>
 <FieldRef Name='Role' />
 <Value Type='Text'>Project Manager</Value>
 </Eq>
 </And>
 </Where>
 </Query>
 <ViewFields>
 <FieldRef Name='ProjectID'/>
 <FieldRef Name='Role'/>
 </ViewFields>
 </View>" 

 $Items = $List.GetItems($Query)

Summary

BCS has been around for a while and isn’t something very existing. But as mention in the introduction, using it for data migrations is very powerful, specially when you don’t have any 3. parts tools around. Doing this from PowerShell is pretty straight forward, but these three issues I experienced can be a headache if you get stuck with them. Hopefully this post can help others get quickly passed them!

Export-SPWeb fails with “These columns don’t currently have unique values”

During a content database migration from SharePoint 2010 to 2013, we also had a requirement to move a few sites (SPWeb) to new locations. The plan was to first mount the Content Database on the SharePoint 2013 farm, create an evaluation upgrade site collection and then export the content from the upgraded site using “Export-SPWeb”. This operation failed, and after several retries it seems like this is not supported.

PowerShell command to export site:

Export-SPWeb http://sps:8080/ -Path "C:\Backup\sps-8080.bak"

This error was found in the log file:

[13.01.2014 10:26:58] FatalError: These columns don't currently have unique values.
[13.01.2014 10:26:58] Debug:    at System.Data.ConstraintCollection.AddUniqueConstraint(UniqueConstraint constraint)
   at System.Data.ConstraintCollection.Add(Constraint constraint, Boolean addUniqueWhenAddingForeign)
   at System.Data.ConstraintCollection.Add(Constraint constraint, Boolean addUniqueWhenAddingForeign)
   at System.Data.DataRelationCollection.DataSetRelationCollection.AddCore(DataRelation relation)
   at System.Data.DataRelationCollection.Add(DataRelation relation)
   at System.Data.DataRelationCollection.Add(String name, DataColumn parentColumn, DataColumn childColumn)
   at Microsoft.SharePoint.Deployment.ListObjectHelper.GetNextBatch()
   at Microsoft.SharePoint.Deployment.ObjectHelper.RetrieveDataFromDatabase(ExportObject exportObject)
   at Microsoft.SharePoint.Deployment.ListObjectHelper.RetrieveData(ExportObject exportObject)
   at Microsoft.SharePoint.Deployment.ExportObjectManager.GetObjectData(ExportObject exportObject)
   at Microsoft.SharePoint.Deployment.ExportObjectManager.MoveNext()
   at Microsoft.SharePoint.Deployment.ExportObjectManager.ExportObjectEnumerator.MoveNext()
   at Microsoft.SharePoint.Deployment.SPExport.SerializeObjects()
   at Microsoft.SharePoint.Deployment.SPExport.Run()
[13.01.2014 10:26:58] Progress: Export did not complete.

Solution

This problem seems to occur once you have created an evaluation site within the same content database. Deleting the evaluation site does not fix the problem unfortunately.

  • Restore site collection from backup to a new content database
  • Export data with Export-SPWeb from the site, but avoid using an evaluation site.

It is possible to run Export-SPWeb both when the site collection is in 2010-mode and naturally after upgrading it to 2013 as long you stay away from creating an evaluation site.

Summary

If you plan to reorganize the content in the same operation as a content database migration from SharePoint 2010 to 2013, avoid using an evaluation site as it leaves your entire site collection in a state where content cannot be exported. With this experience I now always take a extra site collection backup before using creating evaluation sites (which itself is a excellent feature).

Preparing the users MySite after a upgrade from SharePoint 2010 to 2013

A common part of a migration from SharePoint 2010 to 2013, is to include the MySite user profile and personal site collections. Both the User Profile and Managed Metadata service applications must be migrated first. After the web application has been created, and the content database(s) has been reattached, the MySite host must be upgraded to 2013. This can easily be done either from the web page or with PowerShell. All customization to the branding will be lost as the Master Page is reverted to “mysite15.master”, and you will be required to create a new Master Page based on this one to keep a custom branding. In 2013 the user will now be presented with a dialog with 1-2 options the first time they visit their MySite after the upgrade. To avoid unnecessary interruption and confusion, I think it is a good think to prepare the MySite so these choices is already set for the user from a company policy.

What will it look to the end users?

Scenario 1: The user only has a user profile, and no personal site collection

mysite 2

This gives the user the option to have some of the options regarding the social features in their user profile activated. It could be better to set this for all users, and give them a guide how to changes it afterwards instead. Most people won’t care, and keep the default settings.

Scenario 2: The user had both  a user profile and a personal site collection

mysite 1

In the last option, SharePoint has detected that the user has a site collection, and that one or more document libraries exists within it. Keeping this option without being aware of what it does can end in trouble.

For a personal site collection with no customization, this probably will work out fine. But if you have created personal document libraries, or even having custom solutions with their own document libraries, this must be handled differently.

What happens if I choose “Ok” (as ALWAYS)?

If the document library “Shared Documents” exists, it is automatically mapped to the folder “Shared with everyone”. All other document libraries are created as new folders. A few libraries are always ignored; Style Library, SiteAssets and FormServerTemplates.

For testing purposes it is possible to run the initial setup multiple times as long you clean up the “Documents” library to only contain the “Shared with everyone” folder.

After the files have been moved to SkyDrive, the original library is removed. SkyDrive itself is stored in the “Documents” library with the hard coded URL “/Documents”.

Permissions are not copied, so if you had libraries, folder or files with unique permissions set, these must be reapplies manually after the merge.

It can be a bit tricky to test and get a good understanding of what will happen, but luckily this one-time dialog can be open up and reapplied as many times you want by using this URL: http://mysite/_layouts/15/InitialSetup.aspx?IsDlg=1&HasMysite=1

To simulate the last option in the dialog box, switch the query attribute “HasMysite” between 0 and 1.

Disabling the default “Let’s get social” dialog

Found a hint at this blog, but it didn’t for some reason work for me: http://www.ilovesharepoint.com/2013/03/get-rid-of-mysite-lets-get-social-dialog.html.

By adding the value to the AllProperties, instead of Properties bag of the SPWeb object as suggested in the article above, the dialog was suppressed.

$web = Get-SPWeb http://mysite/
$web.AllProperties["urn:schemas-microsoft-com:sharepoint:portal:profile:SPS-O15FirstRunExperience"] = "Off";
$web.Update();

Just as a reminder when you are removing this dialog, you should make sure the default MySite configuration that is correct for your organization.

Enable social data for existing users

For all new users, these settings are configured and managed by the service application. For existing users, a policy to enable social data can be applied with PowerShell.

mysite 3

$site = Get-SPSite -Limit 1
$context = Get-SPServiceContext $site
$profileManager = New-Object Microsoft.Office.Server.UserProfiles.UserProfileManager($context)
$profiles = $profileManager.GetEnumerator()
$profiles | where {  $_.ShareAllSocialData($true); }

Summary

After migrating MySite from SharePoint 2010 to 2013, new features including Social and SkyDrive requires end-users to take action. This article discusses what these options include, and how you could set the policy up front, and suppress the dialog from appearing at all.

Disable site collection upgrades after migrating from SharePoint 2010 to 2013

When migrating SharePoint from 2010 to 2013, it depend of the migration strategy you choose how the end result will appear for your end users. Two different approaches when migrating to SharePoint 2013 can be:

  1. Migrate the farm, content and solutions from 2010 to 2013
  2. Migrate the farm to 2013, but keep the content and solutions in 2010-mode

Selecting the last approach requires the least effort, and could be preferred for several reasons:

  • Reduce the scope of the migration to only include the farm. Less risk and effort needed.
  • Migrate content and solution in a later stage after the new 2013 farm has been stabilized.
  • Keep solutions who will have an end-of-life in near future without extra effort to migrate it.
  • Customer has no functional requirements to adopt 2013 functionality at the current time for all or parts of the solutions, and only requires a platform upgrade.

Since the site collection administrators can on their own effort start the upgrade of their site, I will cover how to get control of this process by disabling the site collection upgrades. As an farm administrator you can then later re-enable this feature, or perform the migration on behalf of the owners (maybe preferred)-

How will a upgraded site collection in 2010-mode appear?

When visiting a site collection after the platform has been migrated, pretty much nothing has changed (good!) for the end users, except a light pink (not nice!) bar at the top reminding us that this site should be upgraded.

site collection upgrade 1

On the site collection upgrade page, an option to “Try a demo upgrade” is available. By default this request is put into a queue, and processed once each night. A copy of the site collection is created, and the site owner will receive an e-mail with the URL. After a fixed time of 30 days, the test site will be deleted.

The reason why this is running by night, is by my best guess because the source site will throw an error while the creation of the eval site runs. So don’t be too tempted to run this timer job manually if the site is in use!

The messages on the top of the screen will only be visible to the site collection administrators, so the regular users (visitors, members or owners) will not see this at all.

Disable the self-service evaluation

In the “SharePoint 2013 Management Shell” run the following Powershell script.

$siteUrl = "http://sp2013"; # Change this one!
$site = Get-SPSite $siteUrl;
$site.AllowSelfServiceUpgradeEvaluation = $false;

The option to create a evaluation site is no longer available for the site collection. The next step would be to disable the possibility for the site owners at all to perform the upgrade them self.

site collection upgrade 2

Disable the self-service site collection upgrade

In the “SharePoint 2013 Management Shell” run the following Powershell script.

$siteUrl = "http://sp2013"; # Change this one!
$site = Get-SPSite $siteUrl;
$site.AllowSelfServiceUpgrade = $false;

Now both the options are disabled, and the pink bar at the top of the site is also removed.

site collection upgrade 3

What if I want to disable this on all site collections?

If you want to go all-in, this Powershell script disables both the evaluation site and self-service upgrade for all site collections within a web application:

$webAppUrl = "http://sp2013"; # Change this one!
Get-SPSite -Limit All -CompatibilityLevel 14 -WebApplication $webAppUrl | % { $_.AllowSelfServiceUpgrade = $false; $_.AllowSelfServiceUpgradeEvaluation = $false; }

Summary

In this article we have seen how a site collection appear to the end-users after the farm has been migrated from SharePoint 2010 to 2013, and the content databases attached back on. With a few lines of PowerShell the administrator can disable both the ability to evaluate a upgraded site as well as perform the self-service upgrade.