Monitoring Sitecore on AKS – Application Insights Instrumentation for Distributed Tracing and Sitecore Logs Collection

In a microservice architecture like the containerized Sitecore XP1 topology, the ability to track the progression of a single request through different services is very important to investigate problems, optimize our application code and build more reliable services. This method, called distributed tracing, helps to clearly show the dependencies and relationships among various services and components of a system. In complex distributed systems, where each service generates its own logs, it is also important to be able to correlate application logs to specific web request traces. This method, called contextual logging, automates the association of logs to traces and spans and adds new logging properties (like error messages or error classes) in the group of tracing dimensions, that can be used in querying and troubleshooting processes when trying to identify a specific pattern or clue.

In the Azure cloud world, these two methods can be implemented instrumenting the Sitecore services with Application Insights, a feature of Azure Monitor that provides extensible application performance management and monitoring capabilities.

In this blog post, I will describe how to procure Application Insights in an existing AKS cluster and the steps needed to instrument all services of a containerized Sitecore 10.2 XP1 solution to collect distributed traces and correlated Sitecore logs, targeting the latest Application Insights SDK version (2.20).

Vendor Neutrality for Telemetry Instrumentation

All Sitecore XP roles of Sitecore Azure PaaS installations are always been instrumented out-of-the-box with the Application Insights SDK, to collect application logs, performance counters and request traces. With the introduction of the official Sitecore images to run a containerized Sitecore application in a Kubernetes cluster, the Application Insights instrumentation has been removed from the main core services images: Content Delivery, Content Management and Processing. The main goal of this choice was to avoid instrumentation vendor lock-in with Azure, but the removal process has been implemented only partially. Application Insights instrumentation is still included in all xDB and Cortex services and in Identity Server service images, in a deactivated state.

In order to achieve the vendor neutrality goal on the application tracing instrumentation, Application Insights instrumentation should have been removed from all Sitecore images and replaced with a vendor neutral instrumentation solution, like for example OpenTelemetry, an open source project under the Cloud Native Computing Foundation. Many proprietary monitoring platforms, including Azure Monitor Application Insights, are already capable today to ingest OpenTelemetry-based telemetry data.

Any compatible vendor specific instrumentation, like for example Application Insights, should instead be distributed separately from the Sitecore application, in a distinct module.

For an existing containerized Sitecore application running on an Azure Kubernetes Service cluster, it makes sense to continue to instrument the application using the Application Insights SDK. In the next sections, I will describe its procurement in Azure and the steps needed to instrument all Sitecore images using a custom asset module image for the Application Insights instrumentation (shared in the MonitoringSitecoreOnAKS code repository here). The instrumentation module targets version 2.20 of the Application Insights SDK for a containerized Sitecore 10.2 XP1 solution, upgrading the long-time deprecated 2.5.1 version still included in vanilla Sitecore Azure PaaS installations.

Application Insights Resource Procurement

Before instrumenting the Sitecore application, let’s start creating first a new Application Insights workspace-based resource in Azure in the same resource group where Container Insights and its Log Analytics workspace resources were created when enabling monitoring in the AKS cluster. The new Application Insights resource should be created connecting it to the same existing Log Analytics workspace, to keep all collected data in a single consolidated location.

The Application Insights resource can be created manually via the Azure Marketplace, choosing a workspace-based type and selecting the existing Log Analytics workspace resource in the Workspace Details section (otherwise it will default to create a new Log Analytics workspace resource).

The Application Insights resource can also be created automatically via command line using the Application Insights preview commands for Azure CLI, available after installing the Application Insights CLI preview extension executing the following command in Powershell:

az extension add -n application-insights

Once the extension is installed, the Application Insights resource can be created using the az monitor app-insights component create command as in the following example, where --app parameter is used to specify the name of the resource, --location parameter for the Azure resource location, --resource-group parameter for the name of the resource group where the resource should be created and --workspace for the name of the existing Log Analytics workspace resource to link to:

az monitor app-insights component create --app sc102ai --location eastus --resource-group DefaultResourceGroup-EUS --workspace DefaultWorkspace-3bc639b9-228f-40a6-b5d1-055cc84bb656-EUS

Application Insights Instrumentation in Sitecore Images

The code-based instrumentation of Sitecore images with Application Insights SDK is not different from the instrumentation done for Sitecore solutions running on an IaaS/on-prem infrastructure or on a PaaS infrastructure. If you are not familiar with Application Insights in Sitecore at all, I recommend reading this great article written by João Neto, Solution Architect at Sitecore, that describes in details the Application Insights instrumentation setup for an on-premises Sitecore XP installation.

In a Sitecore Azure PaaS installation, the Sitecore services of an XP1 topology are not instrumented with the same telemetry collectors. Requests tracing is performed on the main core Sitecore roles (Content Management, Content Delivery and Processing) only. Dependencies tracing is available on all Sitecore roles except Identity Server role, but it is not enabled by default in any of them. Logs collection is performed using diagnostic traces for all Sitecore roles, except for the Identity Server role where is performed using custom events. Contextual logging is available only for the main core Sitecore roles where requests tracing is enabled.

The instrumentation shared in the asset module image on the MonitoringSitecoreOnAKS repository keeps the same differentiated collection approach, but enables dependencies tracing on all roles to implement distributed tracing across the complex distributed Sitecore architecture.

The file structure of the Application Insights asset module image follows the standard file structure of a module with files organized by role (ie. cm) and type (content and transforms):

/module/
    <role>/
        content/
        transforms/

Kubernetes Secrets and Environment Variables

The Application Insights instrumentation requires the creation of two new secrets in Kubernetes (included in the ai-secrets folder in the MonitoringSitecoreOnAKS repository here), that are referenced as value source for some of the new environment variables defined for each Sitecore service:

• application-insights-connection-string.txt – The value of the connection string of the procured Application Insights Azure resource (ie. InstrumentationKey=00000000-0000-0000-0000-000000000000;IngestionEndpoint=https://eastus-8.in.applicationinsights.azure.com/;LiveEndpoint=https://eastus.livediagnostics.monitor.azure.com/)

• application-insights-instrumentation-key.txt – The value of the instrumentation key of the procured Application Insight Azure resource (ie. 00000000-0000-0000-0000-000000000000).

Application Insights configuration settings and instrumentation keys are applied to the Sitecore services using environment variables defined in each Kubernetes deployment specs files (shared in the ai-sitecore folder in the MonitoringSitecoreOnAKS repository here).

Environment Variables for Main Sitecore Roles (cm, cd, prc)

The following environment variables have been added to the Kubernetes deployment specs of the main core Sitecore roles (Content Delivery service role values are shown below as example):

- name: APPLICATIONINSIGHTS_CONNECTION_STRING
  valueFrom:
    secretKeyRef:
      name: application-insights
      key: application-insights-connection-string.txt
- name: APPINSIGHTS_INSTRUMENTATIONKEY
  valueFrom:
    secretKeyRef:
      name: application-insights
      key: application-insights-instrumentation-key.txt
- name: Sitecore_ConnectionStrings_appinsights.instrumentationkey
  valueFrom:
    secretKeyRef:
      name: application-insights
      key: application-insights-instrumentation-key.txt
- name: SITECORE_SETTINGS_APPLICATIONINSIGHTS.Role
  value: CD
- name: SITECORE_SETTINGS_APPLICATIONINSIGHTS.Tag
  value: Sitecore
- name: WEBSITE_HOSTNAME
  value: Content Delivery

The WEBSITE_HOSTNAME environment variable has been used to define the value of the Cloud_RoleName property in the collected telemetry records, otherwise left blank in a non-PaaS installation.

Environment Variables for xDB and Cortex Sitecore Roles

The following environment variables have been added to the Kubernetes deployment specs of all other Sitecore services, except the Identity Server role (Cortex Processing service role values are shown below as example):

- name: APPLICATIONINSIGHTS_CONNECTION_STRING
  valueFrom:
    secretKeyRef:
      name: application-insights
      key: application-insights-connection-string.txt
- name: APPINSIGHTS_INSTRUMENTATIONKEY
  valueFrom:
    secretKeyRef:
      name: application-insights
      key: application-insights-instrumentation-key.txt
- name: SITECORE_APPSETTINGS_AppInsightsKey
  valueFrom:
    secretKeyRef:
      name: application-insights
      key: application-insights-instrumentation-key.txt
- name: SITECORE_APPSETTINGS_AppInsightsRoleName
  value: Sitecore Cortex Processing 

Environment Variables for Identity Server Role

The following environment variables have been added to the Kubernetes deployment specs of the Identity Server service:

- name: APPLICATIONINSIGHTS_CONNECTION_STRING
  valueFrom:
    secretKeyRef:
      name: application-insights
      key: application-insights-connection-string.txt
- name: APPINSIGHTS_INSTRUMENTATIONKEY
  valueFrom:
    secretKeyRef:
      name: application-insights
      key: application-insights-instrumentation-key.txt
- name: SITECORE_Logging__ApplicationInsights__InstrumentationKey
  valueFrom:
    secretKeyRef:
      name: application-insights
      key: application-insights-instrumentation-key.txt
- name: SITECORE_Logging__Serilog__Properties__Role
  value: Sitecore Identity
- name: SITECORE_Logging__Serilog__Properties__Tags
  value: ""

The SITECORE_Logging__Serilog__Properties__Role environment variable is used to specify the value of the Role property of the Serilog application.

Step-By-Step Installation Instructions

In this section, I describe the step-by-step process to instrument an existing Sitecore 10.2 XP1 solution running in an AKS cluster.

Step 1 – Clone my MonitoringSitecoreOnAKS repository (if you haven’t done it yet).

Step 2 – In the application-insights folder of your local repository, open the .env file and populate the following environment variables:

• COMPOSE_PROJECT_NAME – The name of your project solution to prefix the image name.
• REGISTRY – The domain of your Azure Container Registry (ending with a slash).

Step 3 – Build the application-insights asset module image, executing the following command in Powershell:

docker-compose build

Step 4 – Push the image to the Azure Container Registry resource used by the AKS cluster, executing the following command:

docker-compose push

Step 5 – Add the application-insights asset module image to your Sitecore 10.2 instance images, following the same usual process described in the “Add Sitecore modules” Sitecore documentation page here, deploying the modules files and apply the included configuration transformations. The following code shows an example of Dockerfile specs used to install the Application Insights module and to build the image of one of the xDB Sitecore services:

# escape=`
 
ARG BASE_IMAGE
ARG TOOLING_IMAGE
ARG APPLICATION_INSIGHTS_IMAGE
 
FROM ${APPLICATION_INSIGHTS_IMAGE} as application_insights
FROM ${TOOLING_IMAGE} as tooling
FROM ${BASE_IMAGE}

SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]

# Copy development tools and entrypoint
COPY --from=tooling \tools\ \tools\

WORKDIR C:\inetpub\wwwroot

# Add APPLICATION_INSIGHTS module files
COPY --from=application_insights \module\xdbrefdata\content .\

# Copy APPLICATION_INSIGHTS module transforms
COPY --from=application_insights \module\xdbrefdata\transforms\ \transforms\application_insights\

# Perform APPLICATION_INSIGHTS module transforms
RUN C:\tools\scripts\Invoke-XdtTransform.ps1 -Path .\ -XdtPath C:\transforms\application_insights

I also shared an example of this installation setup to add this module in the application-insights branch of my forked docker-examples Sitecore repository in the custom-images folder here.

Step 6 – Build your custom Sitecore images, using your existing build pipeline (if you have one). If you are using the example that I shared in my forked docker-examples repository, then you can build the custom images executing the following command in the custom-images local folder in Powershell:

docker-compose -f .\docker-compose.xp1.yml -f .\docker-compose.xp1.override.yml build

Step 7 – If you have built your custom images locally, push them to your Azure Container Registry, executing the following command in Powershell:

docker-compose -f .\docker-compose.xp1.yml -f .\docker-compose.xp1.override.yml push

Step 8 – In the k8s\ai-secrets folder of your local repository, populate the two Application Insights secrets files with the instrumentation key value and the connection strings value of your Application Insights resource (already procured manually or automatically using the Azure CLI, as described in the Application Insights resource procurement initial section of this blog post).

Step 9 – Deploy the Application Insights secrets in the sitecore namespace (or the namespace used by the Sitecore services, if named differently) executing the following command in the root k8s folder of your local repository:

kubectl apply -k ./ai-secrets -n sitecore

Step 10 – In the k8s\ai-sitecore folder on your local repository, edit the kustomization.yaml spec file to replace your_registry/<compose_project_name> placeholder with the names of the images built and pushed at step 6 and 7.

Step 11 – Deploy the updated Sitecore deployment specs for all Sitecore services (that contain the Application Insights environment variables definitions) to your AKS cluster, executing the following command in the k8s folder in Powershell:

kubectl apply -k ./ai-sitecore -n sitecore

Once the deployment is successfully complete, Application Insights telemetry data will start to be collected in the Log Analytics workspace of your AKS cluster and will be visualized in the different feature views under the Investigate left navigation section of the Application Insights resource in Azure.

An easy way to validate that the expected telemetry data is being collected is query the data of the Application Insights data tables (requests, dependencies, traces, customEvents, exceptions, performanceCounters) in the Logs query editor, aggregating the data by the cloud_RoleName property or by the Role custom dimension. For example:

requests
| summarize count() by cloud_RoleName

traces
| summarize count() by tostring(customDimensions.Role)

dependencies
| summarize count() by cloud_RoleName

customEvents
| summarize count() by tostring(customDimensions.Role)

exceptions
| summarize count() by tostring(customDimensions.Role)

performanceCounters
| summarize count() by tostring(customDimensions.Role)

Conclusions

In this blog post, I described how to procure a workplace-based Application Insights resource in an existing AKS cluster and the steps needed to instrument all services of an existing containerized Sitecore 10.2 XP1 solution using a custom module asset image targeting the latest version of the Application Insights SDK (version 2.20). The shared Application Insights instrumentation should not be considered final, but as a starting point that can be extended, for example implementing custom telemetry initializers to enrich requests telemetry data with additional custom dimensions.

If you have any questions, please don’t hesitate to reach out or comment on this post. Thank you for reading and stay tuned for the next post!