I wanted to set up a daily Sitecore content synchronization to lower environments to ensure they are updated with the latest content from Production every morning.
The Sitecore Sidekick PowerShell module came to the rescue – Thank you Jeff Darchuk for the excellent module! One of its key advantages is that it eliminates the need to pass usernames or passwords for higher environments.
P.S. I hope you have the Sidekick module installed, it needs a SharedSecret to communicate to the higher environments.
Content Sync:
After successfully installing the Sidekick PowerShell module, I created the following script to sync the Home page and its children from the Production environment to the QA environment.
Similarly, I created the following script for the Data folder on the SXA site, excluding the Sitewide Configuration item.
Scheduling:
With the scripts ready, I have scheduled both using Powershell Scripts Command to run every 12 hours to fetch items.
While working on a Sitecore 10.2.1 XM project, we relied on packaging content from upper environments or restoring the database to pull content locally, which was quite time-consuming.
The Sitecore Sidekick module (a big thanks to Jeff Darchuk) was a lifesaver for quickly pulling content into local or lower environments.
The latest Sidekick version is 1.7 and I was wondering which version will be compatible with Sitecore 10.2.1 and realized it doesn’t matter!
Installation
In NuGet, install the following modules in one of the Foundation projects or create a new helix module.
SitecoreSidekickFoundation
SitecoreSidekickContentMigrator
SitecoreSidekickCore
SitecoreSidekickAuditLog
SitecoreSidekickEditingContext
Note: You can skip AuditLog and EditingContext if don’t want it.
After installation, you will see the following DLLs in the bin folder
Configuration
Create a folder Sidekick under App_Config/Include in the Foundation layer.
Add the following Config files as it is, no changes are needed.
We have some secure PDFs in Media Library that were not getting indexed in Solr – They couldn’t be extracted using the PDFSharp library.
The logs were showing the error while extracting secure files
16804 12:04:53 ERROR DefaultMediaItemTextExtractor: Cannot extract content from media item with id ‘{442006A5-8CB6-4ABE-8855-786D2A870201}’. Exception: PdfSharp.Pdf.IO.PdfReaderException Message: The PDF document is protected with an encryption not supported by PDFsharp. Source: PdfSharp at PdfSharp.Pdf.Security.PdfStandardSecurityHandler.ValidatePassword(String inputPassword) at PdfSharp.Pdf.IO.PdfReader.Open(Stream stream, String password, PdfDocumentOpenMode openmode, PdfPasswordProvider passwordProvider) at PdfSharp.Pdf.IO.PdfReader.Open(String path, String password, PdfDocumentOpenMode openmode, PdfPasswordProvider provider) at Sitecore.ContentSearch.ContentExtraction.Readers.PdfSharpReader.ReadAll(String filePath) at Sitecore.ContentSearch.ContentExtraction.Common.DefaultMediaItemTextExtractor.ExtractTextFromMedia(MediaItem mediaItem)
38536 12:04:53 ERROR DefaultMediaItemTextExtractor: Cannot extract content from media item with id ‘{442006A5-8CB6-4ABE-8855-786D2A870201}’. Exception: PdfSharp.Pdf.IO.PdfReaderException Message: The PDF document is protected with an encryption not supported by PDFsharp. Source: PdfSharp at PdfSharp.Pdf.Security.PdfStandardSecurityHandler.ValidatePassword(String inputPassword) at PdfSharp.Pdf.IO.PdfReader.Open(Stream stream, String password, PdfDocumentOpenMode openmode, PdfPasswordProvider passwordProvider) at PdfSharp.Pdf.IO.PdfReader.Open(String path, String password, PdfDocumentOpenMode openmode, PdfPasswordProvider provider) at Sitecore.ContentSearch.ContentExtraction.Readers.PdfSharpReader.ReadAll(String filePath) at Sitecore.ContentSearch.ContentExtraction.Common.DefaultMediaItemTextExtractor.ExtractTextFromMedia(MediaItem mediaItem)
After the installation is completed open the following URL http://localhost:9998 to see if it is working as expected. You should see the welcome message!
Add the following patch file into App_Config/Include/zzz folder to replace DefaultMediaFileTextExtractor from Sitecore.ContentSearch.ContentExtration.
Last step – Let’s add Tika URL into ConnectionStrings.config file.
The Sitecore Command Line Interface (CLI) is a developer tool that provides a command-line interface to interact with Sitecore instances for serializing templates, layouts, and renderings.
Let’s explore the installation and Configuration on Sitecore 10.x.
To initialize a new project, run the below command.
dotnet sitecore init
Note: Starting from 4.1.0, the sitecore init command takes care of installing default plugins (Sitecore.DevEx.Extensibility.Serialization, Sitecore.DevEx.Extensibility.Publishing, Sitecore.DevEx.Extensibility.Indexing, and Sitecore.DevEx.Extensibility.ResourcePackage).
Skip the below step if you are installing Sitecore CLI 4.1.0 or above.
I installed Coveo 5.0.1039.1 on the Sitecore 10.1 instance locally.
After Coveo activation, the indexes weren’t loading and threw an error.
The logs showed ‘The parameter ‘p_ApiKey’ must not be an empty string’ error.
Exception: System.ArgumentException
Message: Precondition failed: The parameter 'p_ApiKey' must not be an empty string
Parameter name: p_ApiKey
Source: Coveo.Framework
at Coveo.Framework.CNL.Precondition.RaiseArgumentException(String p_Message, String p_ParameterName)
at Coveo.Framework.CNL.Precondition.NotEmpty(String p_Parameter, String p_ParameterName)
at Coveo.CloudPlatformClientBase.Communication.CloudPlatformHttpClientFactory.CreateAuthorizedJsonHttpClient(String p_ApiKey)
at Coveo.CloudPlatformClientBase.CloudPlatformClient..ctor(CloudPlatformConfiguration p_Configuration, ICloudPlatformHttpClientFactory p_CloudPlatformHttpClientFactory, IPipelineRunnerHandler p_PipelineRunnerHandler, ISerializer p_Serializer, ICoveoSettings p_CoveoSettings, IStaticTTLCacheFactory`2 p_StaticTTLCacheFactory, ICriticalExceptionHandler p_CriticalExceptionHandler)
at Coveo.CloudPlatformClientBase.CloudPlatformClient..ctor(CloudPlatformConfiguration p_Configuration)
at Coveo.CloudPlatformClientBase.Communication.CloudPlatformClientFactory.GetCloudPlatformClient(CloudPlatformConfiguration p_Configuration)
at Coveo.SearchProvider.Licensing.CloudLicenseRetriever.GetCloudLicense()
at Coveo.SearchProvider.Licensing.CloudLicenseRetriever.GetLicense(Boolean p_ForceRetrieve)
at Coveo.SearchProvider.Licensing.Cloud.LicenseRetriever.GetLicense(Boolean p_ForceRetrieve)
at Coveo.SearchProvider.Licensing.LicenseManager.RetrieveLicense(Boolean p_ForceUpdate)
at Coveo.SearchProvider.Licensing.LicenseManager.EnsureValidLicense()
at Coveo.SearchProvider.Licensing.LicenseManager.GetLicenseInformation()
at Coveo.SearchProvider.Rest.SitecoreRestHttpHandler.InitializeLicenseSettings()
at Coveo.SearchProvider.Rest.SitecoreRestHttpHandler.OnInitializeSettings()
at Coveo.Search.Api.Proxy.ProxyHttpHandler.OnInitialize()
at Coveo.Search.Api.Proxy.ProxyHttpHandler.EnsureInitialized()
at Coveo.Search.Api.Proxy.ProxyHttpHandler.ProcessRequest(IHttpContext p_Context)
at Coveo.SearchProvider.Rest.SitecoreRestHttpHandlerDispatcher.ProcessRequest(HttpContext p_Context)
at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step)
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
After researching, came to know there are two API Keys in Coveo.CloudPlatformClient.Custom.config needs to match with Coveo Platform.
1. apiKey 2. searchApiKey
When I logged into the platform, the keys were not visible since it was secure (not sure where I saved it!), decided to create new keys.
1. SearchApiKey
To create the Search API Key, we must ensure the correct permissions are in place.
Ensure that Impersonate -> Allowed is selected to limit the scope of the API Key, which can be selected from the drop-down list.
2. ApiKey
To create the ApiKey, we need to set multiple privileges.
Content Tab:
Fields -> Edit
Security Identities -> Edit
Security Identity Providers -> Edit
Sources -> Edit all
Organization Tab:
Organization -> Edit
Search Tab:
Search Page -> View all
When the keys are created, make sure to save them in a secure place!
It is time to update the new config keys.
Modify the apiKey and secureApiKey values in Coveo.CloudPlatformClient.Custom.config under AppConfig/Include/Coveo folder
Let’s reload Coveo Index Manager and no more errors.
Download the Configuration files for upgrade <Sitecore 10.3.0 rev. 008463 (upgrade files).zip>
Extract the Sitecore 10.3.0 rev. 008463 (upgrade files).zip file
Scripts Mapping
Now we bunch of files in the extracted folder. We don’t need to run all scripts, Sitecore has given a chart for each Sitecore version upgrade with a list of scripts to run. I’m upgrading from Sitecore 9.3, below is the list of scripts I need to run against clone databases.
Script A – CMS_core_master_web8x.sql
Script B – CMS_core_master_web9x.sql
Script C – CMS_core.sql
Script D – CMS_master.sql
Script E – CMS_web.sql
Script F – SXP_experienceforms_storage.sql
Script H – CMS_security.sql
Core DB:
Run the Scripts C (CMS_Core.sql) for [Your_Project]_Core_Clone DB
<<CMS_core.sql>>
Run the Script H(CMS_Security.sql) against [Your_Project]_Core_Clone DB since I have a separate Security DB connection string.
<<CMS_security.sql>>
Master DB
Run the Scripts D (CMS_Master.sql) for [Your_Project]_Master_Clone DB
<<CMS_master.sql>>
Web DB
Run the Scripts E (CMS_Web.sql) for [Your_Project]_Web_Clone DB
<<CMS_web.sql>>
Experience Forms DB
Run the Script F (SXP_experienceforms_storage.sql) for [Your_Project]_ExperienceForms_Clone DB
<< SXP_experienceforms_storage.sql s>>
Now we ran all the upgrade scripts, let’s update the connection string of vanilla Sitecore 10.3 XM to Cloned databases.
Update the ConnectionString of Sitecore 10.3 XM Instance:
<<ConnectionStrings.config>>
Let’s point to the Clone Databases by updating Initial Catalog and Password.
Use this tool only if you are upgrading to 10.1.0 or later
The default items have been moved into resource files, the tool will remove them from your databases. (if they have not been modified) from the Core, Master, and Web databases.
If you have modified any of the default items, they are not removed and the tool adds a list of these items to its log file.
Sitecore recommends that you review this list and decide if you still need the modifications, and re-apply your customizations to the new items after you have deleted the old ones.
If you decide that you don’t need the modifications, delete these items from SQL databases.
That default content would come from the actual items resource file which will be placed in App_Data\Items.
If your solution contains any modules or connectors, you should use this tool to upgrade them.
Database Clean-up:
It is a one-time operation, run before running the Sitecore UpdateApp tool
To clean up the content databases:
On the Sitecore Launchpad, open the Control Panel, click Clean up databases, select all the databases, and then click Clean.
It will take some time. Once it succeeds, let’s move to the next step.
I navigated to Sitecore.UpdateApp 1.3.1 page since I’m upgrading it to Sitecore 10.3. The following chart shows which version of the tool you should utilize.
Inside the tool page, you must download the package for the version of Sitecore that you are upgrading from. This file contains the clean-up tool.
Since I’m upgrading from 9.3, I choose Sitecore 9.3 XP.
Download the Sitecore.UpdateApp 1.3.1 for Sitecore 9.3.0 rev. 003498 (XP).zip
Extracted the content to C:\Sitecore_103_Upgrade\Sitecore.UpdateApp
Copy the license file to the Data folder of the tool
2023 has been a great year – I went to India in the Summer and Minneapolis in the Fall to attend three events – MVP Summit, Sitecore DX, and SUGCON- NA (had a lot of learning in organizing!).
Now that Coveo is installed and ready to use, Let us add a Sitecore website to our instance of Coveo.
In the Coveo Admin Console, we can see that there is already a Coveo Master, and Web indexes are added by using the Sitecore Package Installation.
In this blog, I am showing an approach without any installation or configuration on the Sitecore Instance. This approach can be used with any non-Sitecore websites as well.
Let’s add the Sitecore website as a web source of content.
We have lots of options to choose from let us focus on web for now.
Here I’m adding my blog as a Source for our demo.
As soon as adding website name, it recognizes and shows the Sitemap URL and recommends adding it for better performance.
After adding the Sitemap URL, Coveo has a few configurations like Authentication and Content security.
Since the blog is public-facing no security configuration would be needed.
Now add and build the source to populate the index. It might take time depending on the number of pages on the site.
When it’s completed it will turn green and show the number of items in the index and the size.
Now that the Index is built, let’s navigate to Content Browser to see if our pages show up.
Navigate to the content browser and search for a term.
Voila! It works! Now you could integrate Coveo search on your site with a preferred approach.
