In-Place Repository Upgrade Automation In AEM OpenCloud

11 Sep 2019 In-Place Repository Upgrade Automation In AEM OpenCloud

Posted by

Once the decision is made to upgrade AEM, you will face a lot of different challenges and and a lot of questions needs to be answered. One question is which upgrade scenario are you going to choose package export-import or in-place migration? One big challenge is how can you provide your developers with an production like AEM environment running on the upgraded version ? Why should you be worried about it? Because you will never know how your code will react with AEM in this future environment.

Here I am going to show you how we automated the AEM In-Place Upgrade from version 6.2 to 6.4 Service Pack 3 with AEM OpenCloud and its CI/CD tools.

I’m going to use AEM-OpenCloud Manager to provision Jenkins. AEM-OpenCloud Manager helps you to manage your AEM OpenCloud environment by providing ready to use CI/CD pipelines

But before we can even start, we need to make sure the following prerequisites are met:

  • AEM installation including repository as AWS snapshots
  • Generate machine images with Packer-AEM
    • AEM Version (AEM 6.2 SP1)
    • AEM Version (AEM 6.4 SP3)
  • Placing all required artifacts on S3
    • AEM SSL Certificate
    • AEM 6.4 Jar File
    • AEM 6.4 Service Pack 3
    • Public certificate of the SAML Service Provider for enabling SAML authentication

Configuration profiles

The first thing we need to setup are two configuration profiles for our AEM stacks. One will be used to create the upgrade environment and the other one to create our AEM 6.4 SP3 environment. I’m going to use the configuration profiles aem-upgrade-aem62-aem64 and aem-consolidated-rhel7-aem64 from the AEM OpenCloud repository aem-helloworld-config and will explain you the necessary parameters for the upgrade.

We will use the config profile aem-upgrade-aem62-aem64 for the upgrade stack. This config profile enables the installation of the AEM upgrade tools and the repository reconfiguration. Additionally, it will remove the existing AEM Global Truststore and all *.bak files older than 30 days from the repository.

# AEM upgrade configuration profile

aem:  
  # Enable repository reconfiguration
  enable_reconfiguration: true

  # Install AEM Upgrade tools
  enable_upgrade_tools: true

  # Enable cleanup of *.bak files from the repository
  # older than 30 days during offline-compaction
  enable_bak_files_cleanup: true  
  bak_files_cleanup_max_age_in_days: 30

# Repository reconfiguration parameters
reconfiguration:
  # Remove existing AEM Global Truststore
  enable_truststore_removal: true

  # Enabling SSL communication for AEM
  # These parameters are similar to the one used by Packer-AEM
  certs_base: s3://aem-opencloud/certificates
  ssl_keystore_password: changeit

  # Enable the creation of the AEM OpenCloud system users
  enable_create_system_users: true

The configuration profile aem-consolidated-rhel7-aem64 will be used with the default settings to create our first upgraded AEM 6.4 SP3 Stack.

Upgrade steps

Now that all requirements have been met, let’s take a look at the in-place upgrade steps. For those who want to know more about this upgrade process, I recommend you to read the official Adobe AEM Upgrade guideline. But In a nutshell the upgrade steps are:

The automation of these steps with AEM OpenCloud required us to develop the automation tools, new InSpec tests for the the InSpec profile InSpec-aem and new Stack Manager events for AEM Stack Manager to trigger the upgrade steps in Jenkins.

Preparation of the AEM Quickstart jar file

The upgrade step preparation of the AEM Quickstart jar file backups the current crx-quickstart directory, prepares the instance for the upgrade and unpacks the AEM 6.4 jar file.

# java -Xmx4096m -jar aem-quickstart.jar -unpack

After the preparation, ensure that the AEM 6.4 quickstart jar file exists at crx-quickstart/app/cq-quickstart-6.4.0-standalone-quickstart.jar

Content Repository Migration

For the content repository migration. The crx2oak migration tool, defined in the Jenkins parameter source_crx2oak, needs to get prepared. Afterwards we can trigger the content repository migration:

# java -Xmx4096m -jar cq-quickstart-6.4.0-standalone-quickstart.jar -v -x crx2oak -xargs -- --load-profile segment-no-ds

Once the migration is done the logfile crx-quickstart/logs/upgrade.log gets checked for the following keywords:

  • does it contain migration completed ?
  • does it not contain WARN ?
  • does it not contain ERROR ?

If all conditions are true, then the content repository migration is successful and the upgrade can be performed.

Performing The Upgrade

Performing the AEM upgrade is the most complicated and critical part of the upgrade process.

This example performs the AEM 6.4 upgrade on the author instance:

# java -server -Xmx4096m -Dcom.adobe.upgrade.forcemigration=true \
-Djava.awt.headless=true -Dsling.run.modes=author,crx3,crx3tar \
-jar crx-quickstart/app/cq-quickstart-6.4.0-standalone-quickstart.jar start -c crx-quickstart -i launchpad \
-p 4502 -Dsling.properties=crx-quickstart/conf/sling.properties

Unlike the previous upgrade commands, this command does not exit by itself, therefore we are waiting ten minutes before killing the command.

# sleep 600

Our experience shows that a wait time of 10 minutes is enough to finish the AEM upgrade on an AWS m4.4xlarge instance with a repository size of ~30 Gigabyte. This value can be overwritten if necessary.

Afterwards the logfile crx-quickstart/logs/upgrade.log gets checked again for the following keywords:

  • does it contain UPGRADE TASK DONE
  • does it contain UPGRADE FINISHED
  • does it not contain ERROR

Once all conditions are true, then the upgrade is successful and the java process gets terminated, then the regular AEM start gets triggered. Afterwards we are checking the endnote /system/console/status-productinfo.json for the installed AEM version.

All these tests are giving us enough confidence that the AEM upgrade was successful.

AEM Upgrade pipeline

How do we integrate the upgrade steps in our automation process? Since we already have a process for creating an AEM environment, we only need to extend this process with our upgrade steps.

There are two important points. Firstly the offline-compaction will delete all *.bak files from the repository older than 30 days, as we configure this in the configuration-profile earlier. The second important point is we are using the deploy-artifacts feature to deploy three different deployment-descriptor. The first one is deploying custom code compatible for AEM 6.2. The second one is deploying the custom code compatible to AEM 6.4 and maybe some further packages e.g. trigger a repository reindexing. The third one will install AEM 6.4 Service Pack 3.

Here is an example of the AEM Service-Pack 3 deployment-descriptor:

{
  "author-publish-dispatcher": {
  "packages": [
    {
      "source": "s3://aem-opencloud/adobeaemcloud/AEM-6.4.3.0-6.4.3.zip",
      "group": "adobe/cq640/servicepack",
      "name": "aem-service-pkg",
      "version": "6.4.3",
      "replicate": false,
      "activate": false,
      "force": false,
      "sleep_seconds": "900"
    },
    {
      "source": "s3://aem-opencloud/adobeaemcloud/AEM-6.4.3.0-6.4.3.zip",
      "group": "adobe/cq640/servicepack",
      "name": "aem-service-pkg",
      "version": "6.4.3",
      "replicate": false,
      "activate": false,
      "force": false,
      "aem_id": "publish",
      "sleep_seconds": "900"
    }
  ], 
  "artifacts": [
    ]
  }
}

As you can see we are installing the service pack separately on the author and publish instances, and after the installation we are waiting 900 seconds before proceeding with the next step. Based on our experience this should give AEM enough time to install the service pack and be ready afterwards.

Now that we have enough knowledge about the upgrade process and the automation process, let’s start upgrading AEM 6.2 to AEM 6.4 SP3.

Upgrading AEM

Assuming aem-opencloud-manager was already successfully provisioned and all configuration profiles are prepared we can start the upgrade by running the pipeline aem-upgrade-in-place which can be found in the category migration, and fill out all necessary parameters.

AEM upgrade pipeline parameters

Define the snapshot ids of the repository with SOURCE_AUTHOR_SNAPSHOT_ID & SOURCE_PUBLISH_SNAPSHOT_ID or if the repository snapshots are already AEM-OpenCloud compatible you only need to define SOURCE_SNAPSHOT_STACK_PREFIX. The parameter STACK_S3_BUCKET is the bucket name of the AEM Stack and is needed for uploading all three the deployment descriptor files PRE_UPGRADE_DEPLOYMENT_DESCRIPTOR_URL, POST_UPGRADE_DEPLOYMENT_DESCRIPTOR_URL and AEM_UPDATE_DEPLOYMENT_DESCRIPTOR_URL to the right location.

Once the pipeline is triggered it may take 2 to 3 hours until the AEM upgrade is completed. On a stack where the AEM Hello World package is installed it took ~1h 35min.

AEM 6.4 SP3 upgrade pipeline run

Once the pipeline finishes successfully, we will have new snapshots which can be used for all new AEM 6.4 environments that we are going to create.

Create an upgraded AEM stack

When creating a new AEM 6.4 stack we just need to define the parameter SOURCE_SNAPSHOT_STACK_PREFIX with the stack prefix of the automation pipeline e.g. AEMUPGRADE-CON. This will lookup the latest snapshots for this stack and attaches them to the new stack. After the stack creation, we should be able to access our upgraded repository and when we navigate to the AEM welcome page we see that AEM is running on version 6.4.3.0

Successfully upgraded to AEM 6.4 SP3

AEM 6.5 Upgrade

At the time when I started writing this article, AEM 6.5 was not available. But now that AEM 6.5 is available, I’m curious how it works with AEM 6.5 .

So I ran the upgrade automation for upgrading my AEM 6.2 instance to AEM 6.5.

The prerequisites are the same as for AEM6.4 SP3 upgrade and the parameters I supplied to Jenkins are very similar to the ones I provided for the AEM Upgrade to AEM 6.4 SP3 as well. The ones I’ve overwritten are:

AEM_UPGRADE_VERSION: 6.5
SOURCE_CRX2OAK:

Since we want to upgrade to AEM 6.5, we need to replace the AEM_UPGRADE_VERSION with 6.5, and because I couldn’t find a newer version of crx2oak, I will leave this property empty. The repository migration will then fallback to the default crx2oak tool which is provided by the 6.5 Jar file.

Once triggered it takes approximately 1h until the AEM upgrade to AEM 6.5 is finished. For those who are wondering why it’s faster. We aren’t installing any hotfixes or service-packs.

AEM 6.5 upgrade pipeline run

After the successful AEM upgrade we can check that we are now running on AEM 6.5.

Successfully upgraded to AEM 6.5.0

Conclusion

Besides the already implemented upgrade scenario of content export/import we have also implemented the in-place migration. Thanks to the automation processes the AEM upgrade becomes less of a hassle and as you can see, we can easily upgrade to the latest available AEM version. But what does this mean in a real life scenario?

In the real world your are now able to create automation processes which allows you to improve the code upgrade process by testing the latest code commitments from your developers on an upgraded AEM environment from your production system without any human interactions. Further your developers can determine and fix bugs in a shorter period of time which leads to more efficiency as your developers will have more time they can invest for code quality and improvements. On top of that your application team does not have to deal with how-to perform the AEM upgrade in much detail which also leads to saving resources.

To use this pipeline for a production release I would recommend adding some minor additional tasks depending on your need e.g. performing the Pre-Upgrade Maintenance Tasks or some manual tasks. But this should give you an easier understanding of the AEM In-Place upgrade and how an AEM In-Place upgrade can be automated for a production release.

If you have any suggestions or questions feel free to contact us on Github or join our freshly created AEM OpenCloud LinkedIn Group.

Tags:
, , , ,
michael.bloch@shinesolutions.com
No Comments

Leave a Reply