Automating SharePoint Deployments (aka Continuous Integration) by PowerShell

SharePoint has an infrastructure for packaging, deploying, and upgrading with its own meta language (XAML), but often our requirements during deployment and maintenance goes beyond what’s possible in SharePoint’s out-of-the-box IMHO it’s the case with every software that provides such facilities. Fortunately SharePoint relies on PowerShell, a scripting language meant for automating administrative tasks. Since I’m a fan of good tools that let us get rid of non-creative tasks, long ago I decided to build an extendable script that can take care of main scenarios and also takes care of common pitfalls along the way.

The script uses a clean and understandable XML file to let you control the process without modifying the code. The XML file consists of 3 main segments BeforeDeploy, Deploy and AfterDeploy. You just need to put tasks inside these main nodes in whatever order you like and if the script does not understand a task it simply ignores it. This gives you the possibility to use the same XML file for your other configurations without any need to create a separate one. Let’s look at an example.

<Deployment>
  <BeforeDeploy>
    <Provision>
      <Website Description="Home" Template="CMSPUBLISHING#0" Title="Home" Url="http://dev/myapp/en/home"/>
      <Website Description="Products" Template="CMSPUBLISHING#0" Title="Products" Url="http://dev/epro/en/products"/>
      <Website Description="Support" Template="CMSPUBLISHING#0" Title="Support" Url="http://dev/epro/en/support"/>
    </Provision>
    <Features>
      <deactivate Description="MyApp.Support.Pages" Id="2aa7ea41-6b01-45d4-87e0-a8194ea246fb" Url="http://dev/myapp/en/support"/>
      <activate Description="MyApp.Products.Pages" Id="2aa7ea41-6b01-45d4-87e0-a8194ea246fb" Url="http://dev/myapp/en/products"/>
      <deactivate Description="MyApp.Home.Pages" Id="2aa7ea41-6b01-45d4-87e0-a8194ea246fb" Url="http://dev/myapp/en/home"/>
      <deactivate Description="MyApp.PageLayouts" Id="f21423d1-0052-456f-b727-aa7268fa4e52" Url="http://dev/myapp"/>
      <deactivate Description="MyApp.ContentTypes" Id="807d55b5-77fc-423c-bae3-aedd861a96c4" Url="http://dev/myapp"/>
      <deactivate Description="MyApp.SessionSharing" Id="93a36eb1-5948-44ed-82a4-642bc1a4132b" Url="http://dev/"/>
      <deactivate Description="MyApp.UrlRewrite" Id="1f8cdfca-c7a6-4f11-8520-b02459b79479" Url="http://dev/"/>
      <deactivate Description="MyApp.BCS" Id="4d3288a9-ce52-4a0e-af83-b0ff78e1d40e"/>
    </Features>
  </BeforeDeploy>
  <Deploy>
    <Solutions>
      <Solution Force="true" Path="C:\CODE\MyApp\Package\MyApp.wsp" UpgradeExisting="false">
        <WebApplications>
          <WebApplication>http://dev/</WebApplication>
        </WebApplications>
      </Solution>
    </Solutions>
  </Deploy>
  <AfterDeploy>
    <Features>
      <activate Description="MyApp.BCS" Id="4d3288a9-ce52-4a0e-af83-b0ff78e1d40e"/>
      <activate Description="MyApp.UrlRewrite" Id="1f8cdfca-c7a6-4f11-8520-b02459b79479" Url="http://dev/"/>
      <activate Description="MyApp.SessionSharing" Id="93a36eb1-5948-44ed-82a4-642bc1a4132b" Url="http://dev/"/>
      <activate Description="MyApp.ContentTypes" Id="807d55b5-77fc-423c-bae3-aedd861a96c4" Url="http://dev/myapp"/>
      <activate Description="MyApp.PageLayouts" Id="f21423d1-0052-456f-b727-aa7268fa4e52" Url="http://dev/myapp"/>
      <activate Description="MyApp.Home.Pages" Id="2aa7ea41-6b01-45d4-87e0-a8194ea246fb" Url="http://dev/myapp/en/home"/>
      <activate Description="MyApp.Products.Pages" Id="2aa7ea41-6b01-45d4-87e0-a8194ea246fb" Url="http://dev/myapp/en/products"/>
      <activate Description="MyApp.Support.Pages" Id="2aa7ea41-6b01-45d4-87e0-a8194ea246fb" Url="http://dev/myapp/en/support"/>
    </Features>
  <AfterDeploy>
    <IIS>
      <Recycle AppPool="MyApp - Service"/>
      <Recycle AppPool="SharePoint - DEV"/>
    </IIS>
    <WarmUp Url="http://dev/myapp/en/home/pages/welcome.aspx" />
    <WarmUp Url="http://dev/myapp/en/support/pages/welcome.aspx" />
    <WarmUp Url="http://dev/myapp/en/products/pages/welcome.aspx" />
    <TimerJobs>
      <Start JobName="Variations Propagate Page Job Definition" WebApp="http://dev" WaitForCompletion="false"/>
    </TimerJobs>
  </AfterDeploy>
</Deployment>

The above XML tells the script to provision a hierarchy of websites, then deactivate some features and then deploy the solution (which will also retract and remove the solution if it is previously deployed) after the deployment is done, it activates features, recycle two application pools and warms up the SharePoint web application by requesting a few pages and at the end starts a timer job. Please note that the order of different elements in the XML file defines the order of the tasks to be run. You can save the XML in a file with arbitrary name and call it like so:

Deploy-SPSolutions c:\solutions.xml -Log "c:\install.log"

You can download the latest version of Deploy-SPSolutions (SharePoint CI) script from GitHub You will also find some samples there.

I’m currently cleaning up the script and putting it in GitHub, but the I always make sure the version on GitHub is reliable enough for everyone to use.

Automating SharePoint Deployments (aka Continuous Integration) by PowerShell

How to enable three finger drag-n-drop in Windows 10 on a macbook

In my previous post we saw how easy it is to update a Boot Camp Windows installed on a MacBook to Windows 10, and how you can easily install the latest version of your Windows drivers provided by Apple. In this post I am going to focus on a different issue that annoys all of us who regularly switch back and forth between Windows and OSX, (you guessed it) Trackpad gestures and more specifically three finger drag-n-drop.

One of the features you’ll miss when you switch from OSX to Windows is the ability to drag with three fingers. You can still enable One-Finger-Dragging in the Boot Camp Control Panel (you need to double-click on its icon displayed beside the clock in your taskbar). You just need to go to the Trackpad tab and enable it:

Boot Camp Control Panel - TrackPad - One Finger Dragging

It’s better than nothing, but it’s not good enough, I know. What if we had three finger dragging and all those cool gestures in Windows 10? Is it hidden somewhere? No, it’s not these options are simply not provided by Apple in it’s Trackpad driver for Windows, but don’t be discouraged you still have one possibility at this moment. Some smart guys have developed a free driver called TrackPad++ that makes it possible to have all those gestures and three finger dragging plus many more and it’s free! It’s actually a wrapper around Apple’s native driver.

The real three finger dragging

Basically you need to download and install two small software. The first one changes a setting in Windows registry to let you install unsigned drivers and the second one is the driver itself. Obviously the developers of a free driver cannot afford to pay a fortune to Microsoft on a yearly basis to review and sign their driver! After you install both software you can uninstall the first one without any issue if you don’t like it 😉

Here are the links to these two software:

  1. Power Plan Assistant 3.2a
  2. TrackPad++

Take a look at their website for more information and instructions. Don’t be fooled by a not so beautiful UI in their app, it works like a charm and there is no annoying popup, ad or anything like that! You might even like to keep the first app as well, it will give you some extra options around power management, keyboard’s backlight, etc.

How to enable three finger drag-n-drop in Windows 10 on a macbook

Upgrading from Windows 7 or 8 on Macbook’s Boot Camp to Windows 10

I decided to share my experience of upgrading to Windows 10 so that in case any of you out there who are thinking about upgrading your BootCamp hosted windows to the latest version can be prepared before hand.

Don’t panic!

First of all know in advanced that the upgrade process went smoothly and you’ll still be able to boot either to your OSX or Windows by holding the option (alt) key during startup, so go ahead get busy while reading this.

Where to start? Download, Windows Update or Download

Technically you have two options. First option is to upgrade through Windows Update channel:

picture is taken from here: http://arstechnica.com/information-technology/2015/07/windows-10-is-on-windows-update-now-the-free-upgrades-start-today/

And the second option is to download the installation image and mount it (right click and select Mount in File Explorer) or write it to a DVD. Microsoft has provided a tool called Windows 10 Media Creation Tool. The 32bit edition can be found here and the 64bit edition can be found here. Another option is to directly download the ISO file from this page if you are currently viewing this page in a Mac. MSDN subscribers can download through their MSDN subscriber downloads page.

You can pick the edition you want I leave the researching to you, but it’s more likely that you stay with the similar edition you currently have on your machine.

Setup process

There’s not much to say here. The process couldn’t be easier, everything is done automatically and after the last restart (yes there are a few) you’ll need to pick your wireless if your computer is not connected through a wired link and that’t it. If you are interested in more pictures and description head here.

The after math

So far the experience was not different that a normal PC, but after you log in to your brand new Windows, you’ll discover that not all features of your touchpad is working and because not all the drivers are installed automatically and you’ll have to reinstall all the BootCamp drivers again. You might (like me) think about downloading the drivers manually but according to Apple you won’t get the latest even if you download “Boot Camp Support Software 5.1.5640” you need to download the latest drivers by using Boot Camp Assistant that is already preinstalled in your OSX, it will give you an option to copy all you need to a USB Flash Drive. Then you can reboot to your Windows drive and install the driver package and everything will be back to normal.

Well, that’s about it. How was your experience during and after installation? don’t hesitate to share with in the comments.

Upgrading from Windows 7 or 8 on Macbook’s Boot Camp to Windows 10

Per developer web.config files using out-of-the-box Visual Studio functionality

It might be a repetitive subject and some might argue that it’s even best to have identical web.config files for all developer. But, the fact it sometimes it’s inevitable! you are not always building software from scratch. It might be an extension to a huge software that contains per-machine keys and configurations. Unfortunately there are many blog posts here and there that suggest using batch files and methods that are more like a hack. Here I’m going to explain how to use the out-of-the-box functionality in Visual Studio without any external tool or hack.

If you already know how those two web.debug.config and web.release.config files work you’ll get the idea. Basically they are a kind of Xslt transformation that are selected based on the current configuration (web.($configuration).config) and we want the same thing but based on current username (web.($Username).config). To learn how to use web.config transformations and its syntax visit “http://go.microsoft.com/fwlink/?LinkId=125889

And here is the recipe:

  1. Right-click over the project name in Solution Explorer and select “Add \ New Item…”
  2. Select XML File under Data category and give it a name in the following pattern web.{username}.config (e.g. web.reza.config).
  3. Copy-paste the content of web.debug.config to this new file (i.e. web.{username}.config) to use as a starting point and override the settings as you need. If you don’t feel comfortable with the transformation syntax, take a look at this page. It’s fairly easy.

Here is an example to override the <machineKey> attributes.

<?xml version="1.0"?>
<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
  <system.web>
    <machineKey xdt:Transform="SetAttributes" validationKey="EFEF01678000A361AADF4E01DB5AD356C91111E781660310" decryptionKey="6EF12ECA1A36ACAC4D08212E9FA8F34B1919A3DD818B8E0F" validation="SHA1" />
</configuration>
  1. Save the file.
  2. Right-click over the project name in Solution Explorer and select Unload Project.
  3. Right-click again over the project name and this time select Edit {projectfilename}.
  4. At the end of the file, right before </Project>, paste the following piece of XML (don’t worry I will explain later).
<Target Name="AfterBuild" Condition="Exists('web.$(USERNAME).config')">
    <Copy SourceFiles="web.config" DestinationFiles="obj\$(Configuration)\tempweb.config" />
    <TransformXml Source="obj\$(Configuration)\tempweb.config" Transform="web.$(USERNAME).config" Destination="obj\$(Configuration)\tempweb2.config" />
    <ReadLinesFromFile File="obj\$(Configuration)\tempweb2.config">
      <Output TaskParameter="Lines" ItemName="TransformedWebConfig" />
    </ReadLinesFromFile>
    <ReadLinesFromFile File="web.config">
      <Output TaskParameter="Lines" ItemName="UnTransformedWebConfig" />
    </ReadLinesFromFile>
    <Copy Condition=" @(UnTransformedWebConfig) != @(TransformedWebConfig) " SourceFiles="obj\$(Configuration)\tempweb2.config" DestinationFiles="web.config" OverwriteReadOnlyFiles="True" />
  </Target>
  1. Save the project file.
  2. Right-click on the project file again and select Reload Project this time.
  3. Build and project and check the result.

And now the explanation. Basically what we did was to add an AfterBuild process (you can also use a BeforeBuild) and set a condition for it to only run when there is web.{username}.config file in the project’s root. It means that if there is no such file for the current developer, it will be skipped. Within the process we have 5 actions that run in the order that is written:

  • <Copy> makes a copy of web.config file to “obj\($configuration)” folder with the name “tempweb.config”. The value of $Configuration depends on the build configuration currently selected in your Visual Studio (e.g. Release or Debug).
  • <TransformXml> runs the transformation to override developer specific values in the tempweb.config file and generates another file (tempweb2.config).
  • The two <ReadLinesFromFile> actions load the content of tempweb2.config and original web.config file into two different variables.
  • The last <Copy> replaces the original web.config only if there is any difference between the generated file and the original (Condition="@(UnTransformedWebConfig) != @(TransformedWebConfig)").
Per developer web.config files using out-of-the-box Visual Studio functionality

Preventing IIS time-outs / recycles when debugging your ASP.NET / SharePoint code

If you are developing web applications or SharePoint solutions that are deployed to the front-end you have probably already encountered the following error while you were debugging a long piece of code:

The web server process that was being debugged has been terminated by IIS. this can be avoided by configuring application pool setting in IIS. see help for further details.

You get this error because there is a mechanism in IIS which detects unresponsive web application and tries to restart them. Basically it works by pinging the applications every X seconds (90 by default) and it no answer is received it will recycle them. To give you more time to debug your code you can either disable it to change it to a larger number. I do not recommend disabling it firstly to have a more production like environment and secondly because it is useful!

How to disable or change it to a longer period

  1. Open IIS Manager and go to “Application Pools” node under your web server.
  2. Right-click on your application pool and select “Advanced Settings…”.
  3. Under “Process Model” you will find “Ping Enabled”, “Ping Maximum Response Time” and “Ping Period”.
  4. To disable it, you need to set “Ping Enabled” to “False” and to buy more time you need to set “Ping Maximum Response Time” to a larger number for example 300 could be enough.
Application Pool Advanced Settings
Application Pool Advanced Settings
Preventing IIS time-outs / recycles when debugging your ASP.NET / SharePoint code

Everything you need to know about BCS – Intro

Business connectivity services is an integral part of Microsoft SharePoint Server when it comes to interacting with external systems (i.e. Line of Business Systems). There lots of tools and foundation from QCRUD operations to out-of-the-box web parts to let you surface the data and let users interact with it. For when you need to go further there is a capabale object model that can be utilized to make custom solutions most probably used to develop custom web parts. There is a visual designer in Visual Studio 2010+ that makes it easier to design and validate the interface between SharePoint and your external system. In SharePoint 2013, Microsoft has taken it to the next level by supporting OData, event listeners, apps, REST interface and a better external list (still could be better 😉 ).

With all these said, what makes working wih BCS complex is the shortage of learning resources and the way its object model and BDC files (interface definition) are design (specially the latter). You cannot even trust the documentations on MSDN sometimes (I will come back to this later). Although there are blog posts here and there to help you move forward, sometimes you are on your own to discover hidden gems and pitfalls.

In these series of posts my aim is to share my experience in building real world BCS solutions so you can save valuable development time by focusing on more productive tasks. I will try to show you the safe paths and pitfalls and be as complete as possible. There is not much difference between SharePoint 2010 and 2013, but when things are a little different I will show you how. At the end, comments are always welcome!

Comming up:

  • Part 1 – What can I build and what is the best way to do that?
  • Part 2 – Interacting with databases (SQL Server and Oracle as examples)
  • Part 3 – Interacting with other sources / .NET Connectivity Assemblies
  • Part 4 – Interacting with dynamic sources / Custom Connectors
  • Part 5 – Searchable BCS solutions
  • Part 6 – Enhancing BCS Performance
  • Part 7 – Surfacing BCS with search
  • Part 8 – Surfacing BCS with out-of-the-box web parts
  • Part 9 – Surfacing BCS with custom web parts
  • Part 10 – Surfacing BCS with Java Script and JSON (MVVM) in SharePoint 2010
  • Part 11 – Adding CSOM support to BCS in SharePoint 2010
  • Part 12 – Business Data Catalog file, the complete reference
  • Part 13 – Bullet-proof deployment of BCS solutions in Farm
  • Part 14 – BCS and Claims-Based-Authentication considerations

Do you have any suggestions for another part? then let me know in the comments.

Everything you need to know about BCS – Intro

First step in developing apps for SharePoint Online – Enabling F5

It’s Just a quick post to share one of my first experiences with SharePoint online and that’s how to enable F5 experience in Visual Studio. When you start writing your first App for SharePoint Online you might get the same error as me, saying that “Sideloading of apps is not enabled on this site”

Sideloading of apps is not enabled on this site
Sideloading of apps is not enabled on this site

It’s because the “Developer” feature is not activated by default and you need to activate it manually, but since it’s an online version of SharePoint things are a little bit different that you local development environment (a good article on how to prepare your local development environment) and you need to use CSOM (Client Side Object Model). I have just translated the c# equivalent to a few lines of PowerShell code so you can type it into your SharePoint Online Management Shell (if you have not installed it, you can download it from here). Here is the code:

<br />$siteurl = "https://yourwebsite.sharepoint.com"<br />$username = "yourusername"<br />$password = ConvertTo-SecureString -String 'yourpassword!' -AsPlainText -Force<br />[Microsoft.SharePoint.Client.ClientContext] $context = New-Object Microsoft.SharePoint.Client.ClientContext($siteurl)<br />[Microsoft.SharePoint.Client.SharePointOnlineCredentials] $credentials = New-Object Microsoft.SharePoint.Client.SharePointOnlineCredentials($username, $password)<br />$context.Credentials = $credentials<br />$site = $cc.Site<br />$sideLoadingGuid = New-Object System.Guid "AE3A1339-61F5-4f8f-81A7-ABD2DA956A7D"<br />$site.Features.Add($sideLoadingGuid, $true, [Microsoft.SharePoint.Client.FeatureDefinitionScope]::None)<br />$context.ExecuteQuery();<br />
First step in developing apps for SharePoint Online – Enabling F5