
11 Sep 2019 In-Place Repository Upgrade Automation In AEM OpenCloud
Once the decision is made to upgrade AEM, you will face a lot of different challenges and and a lot of questions needs to be answered. One question is which upgrade scenario are you going to choose package export-import or in-place migration? One big challenge is how can you provide your developers with an production like AEM environment running on the upgraded version ? Why should you be worried about it? Because you will never know how your code will react with AEM in this future environment.
Here I am going to show you how we automated the AEM In-Place Upgrade from version 6.2 to 6.4 Service Pack 3 with AEM OpenCloud and its CI/CD tools.
I’m going to use AEM-OpenCloud Manager to provision Jenkins. AEM-OpenCloud Manager helps you to manage your AEM OpenCloud environment by providing ready to use CI/CD pipelines
But before we can even start, we need to make sure the following prerequisites are met:
- AEM installation including repository as AWS snapshots
- Generate machine images with Packer-AEM
- AEM Version (AEM 6.2 SP1)
- AEM Version (AEM 6.4 SP3)
- Placing all required artifacts on S3
- AEM SSL Certificate
- AEM 6.4 Jar File
- AEM 6.4 Service Pack 3
- Public certificate of the SAML Service Provider for enabling SAML authentication
Configuration profiles
The first thing we need to setup are two configuration profiles for our AEM stacks. One will be used to create the upgrade environment and the other one to create our AEM 6.4 SP3 environment. I’m going to use the configuration profiles aem-upgrade-aem62-aem64 and aem-consolidated-rhel7-aem64 from the AEM OpenCloud repository aem-helloworld-config and will explain you the necessary parameters for the upgrade.
We will use the config profile aem-upgrade-aem62-aem64 for the upgrade stack. This config profile enables the installation of the AEM upgrade tools and the repository reconfiguration. Additionally, it will remove the existing AEM Global Truststore and all *.bak files older than 30 days from the repository.
# AEM upgrade configuration profile
aem:
# Enable repository reconfiguration
enable_reconfiguration: true
# Install AEM Upgrade tools
enable_upgrade_tools: true
# Enable cleanup of *.bak files from the repository
# older than 30 days during offline-compaction
enable_bak_files_cleanup: true
bak_files_cleanup_max_age_in_days: 30
# Repository reconfiguration parameters
reconfiguration:
# Remove existing AEM Global Truststore
enable_truststore_removal: true
# Enabling SSL communication for AEM
# These parameters are similar to the one used by Packer-AEM
certs_base: s3://aem-opencloud/certificates
ssl_keystore_password: changeit
# Enable the creation of the AEM OpenCloud system users
enable_create_system_users: true
The configuration profile aem-consolidated-rhel7-aem64 will be used with the default settings to create our first upgraded AEM 6.4 SP3 Stack.
Upgrade steps
Now that all requirements have been met, let’s take a look at the in-place upgrade steps. For those who want to know more about this upgrade process, I recommend you to read the official Adobe AEM Upgrade guideline. But In a nutshell the upgrade steps are:
The automation of these steps with AEM OpenCloud required us to develop the automation tools, new InSpec tests for the the InSpec profile InSpec-aem and new Stack Manager events for AEM Stack Manager to trigger the upgrade steps in Jenkins.
Preparation of the AEM Quickstart jar file
The upgrade step preparation of the AEM Quickstart jar file backups the current crx-quickstart directory, prepares the instance for the upgrade and unpacks the AEM 6.4 jar file.
# java -Xmx4096m -jar aem-quickstart.jar -unpack
After the preparation, ensure that the AEM 6.4 quickstart jar file exists at crx-quickstart/app/cq-quickstart-6.4.0-standalone-quickstart.jar
Content Repository Migration
For the content repository migration. The crx2oak migration tool, defined in the Jenkins parameter source_crx2oak
, needs to get prepared. Afterwards we can trigger the content repository migration:
# java -Xmx4096m -jar cq-quickstart-6.4.0-standalone-quickstart.jar -v -x crx2oak -xargs -- --load-profile segment-no-ds
Once the migration is done the logfile crx-quickstart/logs/upgrade.log
gets checked for the following keywords:
- does it contain
migration completed
? - does it not contain
WARN
? - does it not contain
ERROR
?
If all conditions are true, then the content repository migration is successful and the upgrade can be performed.
Performing The Upgrade
Performing the AEM upgrade is the most complicated and critical part of the upgrade process.
This example performs the AEM 6.4 upgrade on the author instance:
# java -server -Xmx4096m -Dcom.adobe.upgrade.forcemigration=true \
-Djava.awt.headless=true -Dsling.run.modes=author,crx3,crx3tar \
-jar crx-quickstart/app/cq-quickstart-6.4.0-standalone-quickstart.jar start -c crx-quickstart -i launchpad \
-p 4502 -Dsling.properties=crx-quickstart/conf/sling.properties
Unlike the previous upgrade commands, this command does not exit by itself, therefore we are waiting ten minutes before killing the command.
# sleep 600
Our experience shows that a wait time of 10 minutes is enough to finish the AEM upgrade on an AWS m4.4xlarge
instance with a repository size of ~30 Gigabyte
. This value can be overwritten if necessary.
Afterwards the logfile
gets checked again for the following keywords:crx-quickstart/logs/upgrade.log
- does it contain
UPGRADE TASK DONE
- does it contain
UPGRADE FINISHED
- does it not contain
ERROR
Once all conditions are true, then the upgrade is successful and the java process gets terminated, then the regular AEM start gets triggered. Afterwards we are checking the endnote /system/console/status-productinfo.json
for the installed AEM version.
All these tests are giving us enough confidence that the AEM upgrade was successful.
AEM Upgrade pipeline
How do we integrate the upgrade steps in our automation process? Since we already have a process for creating an AEM environment, we only need to extend this process with our upgrade steps.
- Create an AEM environment
- Run a readiness check
- Pre-deployment of custom code
- Run Offline-Compaction Snapshot
- Preparation of the AEM Quickstart jar file
- Content Repository Migration
- Performing The Upgrade
- Post-deployment of custom code compatible to AEM 6.4
- Deploy AEM6.4 Service Pack 3
- Take offline-snapshot
- Delete the environment
There are two important points. Firstly the offline-compaction will delete all *.bak files from the repository older than 30 days, as we configure this in the configuration-profile earlier. The second important point is we are using the deploy-artifacts feature to deploy three different deployment-descriptor. The first one is deploying custom code compatible for AEM 6.2. The second one is deploying the custom code compatible to AEM 6.4 and maybe some further packages e.g. trigger a repository reindexing. The third one will install AEM 6.4 Service Pack 3.
Here is an example of the AEM Service-Pack 3 deployment-descriptor:
{
"author-publish-dispatcher": {
"packages": [
{
"source": "s3://aem-opencloud/adobeaemcloud/AEM-6.4.3.0-6.4.3.zip",
"group": "adobe/cq640/servicepack",
"name": "aem-service-pkg",
"version": "6.4.3",
"replicate": false,
"activate": false,
"force": false,
"sleep_seconds": "900"
},
{
"source": "s3://aem-opencloud/adobeaemcloud/AEM-6.4.3.0-6.4.3.zip",
"group": "adobe/cq640/servicepack",
"name": "aem-service-pkg",
"version": "6.4.3",
"replicate": false,
"activate": false,
"force": false,
"aem_id": "publish",
"sleep_seconds": "900"
}
],
"artifacts": [
]
}
}
As you can see we are installing the service pack separately on the author and publish instances, and after the installation we are waiting 900 seconds before proceeding with the next step. Based on our experience this should give AEM enough time to install the service pack and be ready afterwards.
Now that we have enough knowledge about the upgrade process and the automation process, let’s start upgrading AEM 6.2 to AEM 6.4 SP3.
Upgrading AEM
Assuming aem-opencloud-manager was already successfully provisioned and all configuration profiles are prepared we can start the upgrade by running the pipeline aem-upgrade-in-place
which can be found in the category migration
, and fill out all necessary parameters.

Define the snapshot ids of the repository with SOURCE_AUTHOR_SNAPSHOT_ID
& SOURCE_PUBLISH_SNAPSHOT_ID
or if the repository snapshots are already AEM-OpenCloud compatible you only need to define SOURCE_SNAPSHOT_STACK_PREFIX
. The parameter STACK_S3_BUCKET
is the bucket name of the AEM Stack and is needed for uploading all three the deployment descriptor files PRE_UPGRADE_DEPLOYMENT_DESCRIPTOR_URL
, POST_UPGRADE_DEPLOYMENT_DESCRIPTOR_URL
and AEM_UPDATE_DEPLOYMENT_DESCRIPTOR_URL
to the right location.
Once the pipeline is triggered it may take 2 to 3 hours until the AEM upgrade is completed. On a stack where the AEM Hello World package is installed it took ~1h 35min.

Once the pipeline finishes successfully, we will have new snapshots which can be used for all new AEM 6.4 environments that we are going to create.
Create an upgraded AEM stack
When creating a new AEM 6.4 stack we just need to define the parameter SOURCE_SNAPSHOT_STACK_PREFIX
with the stack prefix of the automation pipeline e.g. AEMUPGRADE-CON
. This will lookup the latest snapshots for this stack and attaches them to the new stack. After the stack creation, we should be able to access our upgraded repository and when we navigate to the AEM welcome page we see that AEM is running on version 6.4.3.0

AEM 6.5 Upgrade
At the time when I started writing this article, AEM 6.5 was not available. But now that AEM 6.5 is available, I’m curious how it works with AEM 6.5 .
So I ran the upgrade automation for upgrading my AEM 6.2 instance to AEM 6.5.
The prerequisites are the same as for AEM6.4 SP3 upgrade and the parameters I supplied to Jenkins are very similar to the ones I provided for the AEM Upgrade to AEM 6.4 SP3 as well. The ones I’ve overwritten are:
AEM_UPGRADE_VERSION: 6.5
SOURCE_CRX2OAK:
Since we want to upgrade to AEM 6.5, we need to replace the
with AEM_UPGRADE_VERSION
6.5
, and because I couldn’t find a newer version of crx2oak
, I will leave this property empty. The repository migration will then fallback to the default crx2oak tool which is provided by the 6.5 Jar file.
Once triggered it takes approximately 1h until the AEM upgrade to AEM 6.5 is finished. For those who are wondering why it’s faster. We aren’t installing any hotfixes or service-packs.

After the successful AEM upgrade we can check that we are now running on AEM 6.5.

Conclusion
Besides the already implemented upgrade scenario of content export/import we have also implemented the in-place migration. Thanks to the automation processes the AEM upgrade becomes less of a hassle and as you can see, we can easily upgrade to the latest available AEM version. But what does this mean in a real life scenario?
In the real world your are now able to create automation processes which allows you to improve the code upgrade process by testing the latest code commitments from your developers on an upgraded AEM environment from your production system without any human interactions. Further your developers can determine and fix bugs in a shorter period of time which leads to more efficiency as your developers will have more time they can invest for code quality and improvements. On top of that your application team does not have to deal with how-to perform the AEM upgrade in much detail which also leads to saving resources.
To use this pipeline for a production release I would recommend adding some minor additional tasks depending on your need e.g. performing the Pre-Upgrade Maintenance Tasks or some manual tasks. But this should give you an easier understanding of the AEM In-Place upgrade and how an AEM In-Place upgrade can be automated for a production release.
If you have any suggestions or questions feel free to contact us on Github or join our freshly created AEM OpenCloud LinkedIn Group.
No Comments