BEST PRACTICES FOR PAN-OS UPGRADE

Resolution

Best Practices for PAN-OS Upgrade

The following procedure documents best practices for customers who are new to the PAN-OS upgrade process. It’s intended as a
foundation for customers who want to create their own more-specific upgrade procedures.

About the PAN-OS upgrade and customer responsibilities

 You cannot skip installation of any feature release versions in the path to your target PAN-OS
release. Additionally, it is best practice to always download and install the latest maintenance release
for each feature release and then reboot before you install the base image for the next feature
release—this applies to each feature release through which you pass in the upgrade path.

 In this example, we are upgrading a hypothetical customer (ACME, Inc.) from PAN-OS 7.0.16 to 8.0.6-h3 (with 7.1 as an interim step).
 ACME firewall is configured in Active/Passive HA cluster managed by Panorama (this is the most common configuration in use today).
We are not covering Active/Active, non-HA scenarios or scenarios where there is no Panorama installed.
 This is a best practice document. It is not meant to be a step-by-step procedure. Please create your own step-by-step procedure,
if necessary.
 Customer is responsible for verifying all steps before the upgrade.

Terminology
Active firewall The firewall in an HA cluster that’s passing traffic
Passive firewall The firewall in an HA cluster that’s not passing traffic
Primary firewall The firewall in an HA cluster that’s usually the active firewall
Secondary firewall The firewall in an HA cluster that’s usually the passive firewall
Feature release
Contains new features and bugfixes, typically ends with .0 (i.e. 7.1.0)
Maintenance release Bug fixes only, typically ends with .number (7.1.2)

Dependencies
 Before upgrade, make sure the firewall is running a version of app + threat (content version) that meets the minimum requirement of the
new PAN-OS (see release notes: https://www.paloaltonetworks.com/documentation.html). We recommend always running the latest
version of content to ensure the most accurate and effective protections are being applied.
 Panorama should be running the same or a later version of a feature release than the firewall (up to two feature versions is supported but
not recommended as of June 2016).

If the firewall is running Panorama versions supported are

PAN-OS 7.0.x 7.0.x, 7.1.x, 8.0.x
PAN-OS 7.1.x 7.1.x, 8.0.x
PAN-OS 8.0.x 8.0.x

Table of contents

1. Determine the need to upgrade

2. Pre-upgrade checklist
3. Panorama upgrade procedure
4. Firewall upgrade procedure (HA)
5. Post-upgrade checklist
6. Troubleshooting resources
7. Downgrade procedure

1. Determine the need to upgrade

 In most cases, upgrade should be considered for only the following reasons:
o New features that are not available in current version
o Patches for security vulnerabilities in PAN-OS (see Security Advisories page at https://securityadvisories.paloaltonetworks.com)
o Bug fixes that are not available in current version
o Current version is going to End of Life soon (see PAN-OS EOL policy at https://www.paloaltonetworks.com/services/support/end-of-
life-announcements/end-of-life-summary)
 We highly encourage customers to consult with Palo Alto Networks account team for upgrade decision. Your Palo Alto Networks
account team can provide you with a recommended PAN-OS version.
 For purpose of this document, we will be upgrading from 7.0.16 to 8.0.6-h3 to demonstrate upgrading process across two major releases
(7.0 > 7.1 > 8.0).
 NOTE:
For any PAN-OS version prior to PAN-OS 8.0 (so PAN-OS 7.1 and lower) it is recommended to go to the latest maintenance release to
prevent running into snags or issues during the upgrade.
 Note about PAN-OS 8.0 base installation:
For PAN-OS 8.0, The additional (Optional when installing from the updates server. When installing from a manual file this step is
mandatory) step of installing and rebooting the base image was added to accomodate the larger size of the base image. This is
considered best practice.

 HA Upgrade NOTE:
When upgrading across two major release versions at a time, there will be a time when there will be a network outage. Whereas if the
devices are upgraded one major version at a time, HA will remain active, continue to synchronize sessions, and no network outage will
be seen.

To maintain HA sync and activity, upgrade the HA pair in tandem one major release at a time. If you upgrade one device by two major
upgrades, the newly upgraded device will stay in suspended mode with the error peer OS too old. So when you go to start the first
OS upgrade on the second HA device, you will lose network connectivity until the upgrade is completed and the first device is moved out
of suspended mode and into passive mode and HA capabilities resume functioning.

2. Pre-upgrade checklist

 Review release notes.

 Do not schedule Panorama and firewall upgrades at the same time.
 Upgrade Panorama first, wait at least 24 hours and then upgrade the firewall.
 A pro-active support case may be desirable for some critical environments, Please review this article: When to request a Support Event
 Upgrade should be carried out during non-business hours to minimize impact.
 Allocate sufficient time in the change window for upgrade, troubleshooting and possible downgrade procedures. It may take up to 2-3
hours to upgrade a slower/older system, depending on config. Multiply if upgrading across multiple versions.
 Identify business contacts who will help verify application and network functionalities after the upgrade.
 Back up configuration and device state before upgrade.

o Device > Setup > Operations > Save Named Configuration Snapshot
o Device > Setup > Operations > Export Named Configuration Snapshot
o Device > Setup > Operations > Export Device State

o Device > Support > Generate Tech Support File

 Document any non-standard settings that should be applied post-upgrade
o Disable tcp state checking
o Non-syn tcp reject
o Any debug setting if existing (not recommended)
 Stage/Download necessary PAN-OS images ahead of time. You will need both the base image and the latest maintenance release. The
base image should be installed but not rebooted. In this case, we need to download the following versions
o 7.0.18 (it is recommended to bring your current Feature release to the latest recommended maintenance release before preceding)
o 7.1.0 (base image) (NOTE: If you are on a maintenance release version earlier than 7.0.6, you must install 7.0.6 before 7.1.0 will show
up on your software download page)
o 7.1.14
o 8.0.0 for firewalls, 8.0.2 for Panorama (base image) (NOTE: the 8.0 base image and maintenance versions will not become visible in the
download section until you have a version of 7.1 installed)
o 8.0.6-h3
 Following the PAN-OS upgrade, you may need to upgrade associated software (such as Global Protect agent or User ID agent)
o See the Associated Software Versions chart in the release notes
 (Optional but recommended) Arrange for Out-of-Band access (Console access) to the firewall if possible. This is to help recover from
any unexpected situations where we lose connectivity to the firewall after upgrade.

3. Panorama upgrade procedure

 Make sure no policy or configuration changes are being made by acquiring a config lock


 Click on padlock icon on upper right hand corner of GUI
 If there are any locks, please remove the locks or talk with the admin who placed the lock in place, and remove or commit..
 Clear or complete any pending commit job making a commit to panorama before starting the upgrade
 (Optional but recommended) Post-upgrade failover testing:
 Suspend Secondary Panorama to fail connection back to Primary Panorama to make sure failover still works after upgrade.
CLI:

> request high-availability state suspend

GUI:
Device > High Availability > Operations > click Suspend local device.

 Verify suspend status on Secondary Panorama

 Verify all firewalls are connected to Primary Panorama
 Re-enable Secondary Panorama and High Availability function
CLI:

> request high-availability state functional

GUI:
Device > High Availability > Operations > click Make local device functional.

 Primary Panorama Upgrade procedure:

1. Disable Pre-emption if enabled.
(Disable preemption on High Availability settings to avoid unexpected failovers. Disabling preempt configuration change must be
committed on both peers. Once completed, re-enabling must be committed on both peers).
2. Go to Device > High Availability > Election Settings and uncheck Preemptive. Then, commit the change:

3. Suspend Primary Panorama to make Secondary Panorama as active

From Primary Panorama, suspend High Availability function:
CLI:

> request high-availability state suspend

GUI:
 Device > High Availability > Operations > click on Suspend local device.

4. Verify suspend status on Primary/passive Panorama.

5. Verify all firewalls are connected to Secondary/active Panorama.
6. On the Primary Panorama, download, install and reboot 7.0.18
7. Download and install 7.1.0 (base version).
8. Download and install 7.1.14, and reboot to complete the upgrade.
9. Save/export tech support and Device state and save named device config snapshots (this is in case downgrade is needed).
10. Download 8.0.2 (base version) (Recommended) Install the 8.0 base image and reboot before you install the target maintenance release.
11. Download and install 8.0.6-h3, and reboot to complete the upgrade.
12. Save/export tech support and Device state and save named device config snapshots (this is in case downgrade is needed).
13. Re-enable Pre-emption, if so desired.
14. This concludes upgrade on Primary Panorama.
 Secondary Panorama Upgrade procedure
o Download, install, and reboot 7.0.18
o Download and install 7.1.0 (base version)
o Download and install 7.1.14, and reboot to complete the upgrade.
o Save/export tech support and Device state and save named device config snapshots (this is in case downgrade is needed)
o Download 8.0.2 (base version) (Recommended) Install the 8.0 base image and reboot before you install the target maintenance release..
o Download and install 8.0.6-h3, and reboot to complete the upgrade.
o Save/export tech support and Device state and save named device config snapshots (this is in case downgrade is needed)
o This concludes upgrade on Secondary Panorama
o Backup config and device state files just in case:
https://live.paloaltonetworks.com/t5/Configuration-Articles/Back-Up-Configuration-and-Device-State-from-the-CLI/ta-p/56734

(Optional but recommended) Post-upgrade verification

 Verify connectivity between Panorama and Firewalls. If something is not working, skip to troubleshooting section
(For example, check if Panorama is receiving logs with correct time stamp from firewalls after upgrade is completed)
 Test commit-all operations to managed devices, and verify new changes are applied as expected locally on the devices.

4. Firewall upgrade procedure (HA)

It is recommended to upgrade the Primary firewall first and then upgrade the Secondary firewall. This is done for two reasons:
1.) Ensure that HA failover is functioning properly and
2.) Ensure that the passive firewall is functioning properly and is able to pass traffic without issues.

 Disable Pre-emption if enabled. Disable preemption on High Availability settings to avoid unexpected failovers. Disabling preempt
configuration change must be committed on both peers. Likewise, once completed, re-enabling must be committed on both peers.
To disable: Go to Device > High Availability >General > Election Settings <hit edit> and uncheck Preemptive.
Then, perform a commit.
 NOTE: This procedure relies on the administrator having foreseen access to their devices at all times, either by being local or having
OOB connectivity to the management network which is best practice when upgrading the firewall. In the case where you do not have the
option of achieving either, it is a good idea to change the procedure slightly to ensure you dont lose connectivity at the cost of having a
less rigid upgrade path.

Having the preempt enabled will require you to keep this config change in mind during the whole process as it could unexpectedly switch
over the active membership while upgrading.

 Primary firewall Upgrade procedure

1. On Primary firewall, Suspend Primary firewall to make Secondary firewall active
CLI
 > request high-availability state suspend

GUI
Device > High Availability > Operations > click Suspend local device.
NOTE: This will cause an HA failover. We recommend doing this first to verify the HA functionality is working before initiating the
upgrade. Production traffic is now going through the Secondary firewall which is now active.

2. Ask your business owners to verify all applications are working on the network. If there is a problem, skip to troubleshooting
section. If there is any problem, fix it before proceeding with upgrade.
3. Upgrade Primary firewall. You can do this by either directly downloading and installing software onto the firewall itself or via
Panorama Device Deployment > Software option.
4. Download, install and reboot 7.0.18
5. Download and install 7.1.0 (base version).
6. Download and install 7.1.14, and reboot to complete the upgrade.
7. Save/export tech support and Device state and save named device config snapshots (this is in case downgrade is needed).
8. Download 8.0.0 (base version) (Recommended) Install the 8.0 base image and reboot before you install the target maintenance release..
9. Download and install 8.0.6-h3, and reboot to complete the upgrade.
10. On the Primary firewall, verify auto commit completes successfully (FIN OK) by running the command before proceeding to the next
step:

> show jobs all

Run the following command to make Primary firewall functional again:

> request high-availability state functional

11. This concludes upgrade on the Primary firewall.

 Secondary firewall upgrade procedure:


1. Suspend Secondary firewall to make Primary firewall active.
From Secondary firewall, suspend High Availability function
CLI:

> request high-availability state suspend

GUI:
Device > High Availability > Operations > click Suspend local device.

Note: This will cause an HA failover. Production traffic is now going through Primary firewall with new software installed.

2. Ask your business owners to verify all applications are working on the network. If there is a problem, skip to troubleshooting
section. If there is any problem, fix it before proceeding with upgrade.
3. Upgrade Secondary firewall. You can do this by either directly downloading and installing software onto the firewall itself or via
Panorama Device Deployment > Software option
4. Download, install and reboot 7.0.18
5. Download and install 7.1.0 (base version)
6. Download and install 7.1.14. reboot to complete the install
7. Save/export tech support and Device state and save named device config snapshots (this is in case downgrade is needed)
8. Download 8.0.0 (base version) (Recommended) Install the 8.0 base image and reboot before you install the target maintenance release.
9. Download and install 8.0.6-h3. reboot to complete the install
10. Verify auto commit completes successfully (FIN OK) by running the command before proceeding to the next step:

> show jobs all

Run the following command to make Secondary firewall functional again:

> request high-availability state functional

11. This concludes upgrade on the Secondary firewall

 (Optional but recommended) Arrange for Out-of-Band access (Console access) to the firewall if possible.
This is again to help recover from any unexpected situation where we are unable to login to the firewall.

 Backup config and device state files just in case

 Make sure no policy or configuration changes are being made by acquiring a config lock
o Click on padlock icon on upper right hand corner of GUI
 Make sure no pending commit jobs on firewall
 (Optional but recommended) Post-upgrade verification
o Now that both Primary and Secondary firewalls are both running the new software, it’s a good idea to test failover functionality again.
o Run the following comma to suspend Primary firewall to fail traffic to the Secondary firewall
> request high-availability state suspend
o Ask your business owners to verify all applications are working on the network through the Secondary firewall. If there is a problem,
skip to troubleshooting section
o Run the following CLI command to make Primary firewall functional again:

> request high-availability state functional

o Repeat the process to verify traffic works fine through Primary firewall (suspend the Secondary firewall, test functionality on Primary
firewall, then re-enable Secondary firewall)
o This concludes failover test
 (Optional but recommended) Enable preemption if it was disabled due to upgrade
o Re-enabling preempt configuration change must be committed on both Likewise, once completed, re-enabling must be committed on
both peers.
o Go to Device > High Availability > Election Settings and check Preemptive. Then, perform a commit.

 This completes upgrade on the HA pair.

5. Post-upgrade checklist
The following Post-Implementation Activities should be performed prior to the change window end time. Performing these Post-
Implementation Activities prior to the change window end time allows time to complete any potential corrective action that might be
required after performing these activities.

 Compare Post-Implementation results with Pre-Implementation results

o Reapply the non-persistence settings from the pre-checklist
o Check system logs for any unexpected errors
o Check traffic logs for any traffic that’s unexpectedly allowed or denied
o If applicable, verify connectivity and network functionality between firewall and panorama, specifically make sure log forwarding from
firewall to Panorama is working. Also, validate changes will commit between Panorama and the managed devices.
 Check system reports and incidents for any relevant outages.
o Monitor helpdesk tickets post upgrade, this may take 24 to 48 hours
 Let administrators know about the completion of the change.
o Contact all stake holders to communicate any change IDs, describe change activity results, and verify that there are no relevant network
alarms, incidents, or outages.
o Monitor the appliance for any anomalies.

6. Troubleshooting resources

 If business application no longer works after upgrade (reference the links below)
o PAN-OS Upgrade or Content Update Failure https://live.paloaltonetworks.com/t5/Management-Articles/PAN-OS-Upgrade-or-Content-
Update-Failure/ta-p/55127
o Content Version Error Upgrading Major Platform OS https://live.paloaltonetworks.com/t5/Management-Articles/Content-Version-Error-
Upgrading-Major-Platform-OS-with-an-Older/ta-p/54308
 If the device fails to complete auto-commit (reference the links below)
o How to Determine When Auto-Commit is Complete https://live.paloaltonetworks.com/t5/Learning-Articles/How-to-Determine-When-
Auto-Commit-is-Complete/ta-p/61409
 If software fails to install (reference the links below)
o Software Install or Download Push from Panorama to Device will not Complete https://live.paloaltonetworks.com/t5/Management-
Articles/Software-Install-or-Download-Push-from-Panorama-to-Device-will/ta-p/57062
o Commit finishes with an error response https://live.paloaltonetworks.com/t5/Management-Articles/Commit-finishes-with-an-error-
response-cfgpush-s1-dp1-comm-cfg/ta-p/65742
o Failed to Install Licenses https://live.paloaltonetworks.com/t5/Management-Articles/License-Error-quot-Failed-to-Install-Licenses-
Unexpected-Error/ta-p/53219
 If software fails to download (reference the links below)
o Error downloading 7.0.4 https://live.paloaltonetworks.com/t5/General-Topics/Error-downloading-7-0-4-with-7-0-0-previously-
downloaded/m-p/71632/highlight/true#M40858
o Software Download Error:https://live.paloaltonetworks.com/t5/Management-Articles/Software-Download-Error-Failed-to-download-
due-to-server-error/ta-p/57458
 Panorama checklist: (reference the link below)
o https://live.paloaltonetworks.com/t5/Learning-Articles/Quick-Reference-Guide-Helpful-Commands/ta-p/56511?attachment-id=788
o Panorama Connectivity issues: https://live.paloaltonetworks.com/t5/Management-Articles/Troubleshooting-Panorama-Connectivity/ta-
p/54224
 If issues cannot be resolved
o Contact Palo Alto Networks TAC using proactive case number.
o Save configurations of affected network devices.
o Save configurations of the Palo Alto Network devices.
o Add to pcaps, configurations, techsupport files, and logs from near by networking devices for post mortem and troubleshooting by
support teams.
o Go to the Downgrade/back out procedure section

7. Downgrade procedure
If the issue cannot be resolved within the allotted change window, you should revert all changes.

o Verify 7.1.0 (base image version) is still present on the system
o Verify and install 7.1.14. reboot to complete the install. When prompted, use 7.1.14 snapshot file saved during the upgrade.
o Verify 7.0.1 (base version) is still on the system
o Download and install 7.0.18, and reboot to complete the downgrade. When asked, use 7.0.18 snapshot file saved before the upgrade.
Note: After the Secondary firewall is rebooted, the CLI prompt should show non-functional.
o On the prrimary firewall, verify auto commit completes successfully (FIN OK) by running the command before proceeding to the next
step:

> show jobs all

Run the following command to make it functional again:

> request high-availability state functional

o This concludes downgrade on the Primary firewall.
 Downgrade the Primary firewall
o Suspend the prrimary firewall to fail traffic to Secondary
o On the Primary device, suspend the unit from the CLI. Run the command:

> request high-availability state suspend

 Save the following files for analysis:

o Save and export Tech support and device state from both active and passive firewalls
o All core dump files if there is any
o Packet capture of problem traffic, if observed
 Assuming the firewall is currently running 8.0.6-h3 already and traffic is currently going through the Primary firewall, follow the same
upgrade process but in reverse order.
 Downgrade the Secondary firewall
o Verify 7.1.0 (base image version) is still present on the system
o Verify 7.1.14 is still present on the system and install, reboot to complete the install. When prompted, use 7.1.14 snapshot file saved
during the upgrade
o Verify 7.0.1 (base version) is still on the system
o Download and install 7.0.18. reboot to complete the install. When prompted, use 7.0.18 snapshot file saved before the upgrade.
Note: After the Secondary firewall is rebooted, the CLI prompt should show non-functional.
o On the Secondary firewall, verify auto commit completes successfully (FIN OK) by running the CLI command before proceeding to the
next step:

> show jobs all

Run the following command to make it functional again:

> request high-availability state functional

o This concludes downgrade on the Secondary firewall

 (Optional but recommended) Enable preemption if it was disabled due to upgrade.
o Re-enabling preempt configuration change must be committed on both Likewise, once completed, re-enabling must be committed on
both peers.
o Go to Device > High Availability > Election Settings and check Preemptive. Then, perform a commit.

 (Optional but recommended) Ask your business owners to verify all applications are working on the network. If there is a problem,
skip to troubleshooting section.
 Upload all files to the Palo Alto Networks proactive support case for troubleshooting later.
 This concludes the downgrade process.