Tips for Developing a Successful Hardware Refresh Strategy

Introduction

I’ve done a ton of hardware refreshes and I’ve documented the steps below. Hopefully this will help you with your own hardware refresh strategy.

This is focused on doing life cycle management of aging network equipment, but a lot of the steps are general enough it can be applied to most other things as well. In here, we’re assuming you’ve already decided that you need to do a hardware refresh, and you already know which hardware to purchase.

If you’re not sure if you need to do a hardware refresh, we have a page explaining the benefits and challenges of doing so here or you can read our article Signs You Need A Hardware Refresh for more info.

Planning and Preparation

Scope Gathering

This is a simple step that is often overlooked. Make sure that you and your team’s understanding of what needs to be done aligns with the actual needs. This will help ensure that we’re not wasting our time planning for the wrong things.

One way I accomplish this is by typing up the goal and requirements somewhere where everyone can view it. I then review this with everyone involved and ask if there’s anything I missed or misunderstood.

Start breaking down everything that needs to be done for your hardware refresh strategy into individual steps. You can use this to create sub-tasks to allow for collaboration/delegation with your team.

Also having a checklist reduces mistakes and makes sure that nothing gets skipped.

You’ll need to do some legwork here yourself, but hopefully we’ve already done some of the work for you below.

Build Timeline

Determine a rough schedule of what is being replaced when. Ideally, you would start with the lowest risk devices and end with the highest risk areas.

Try to avoid doing everything at once. Instead of doing everything at once over the weekend, it’s better to break it down by device cluster/stack each change window.

Benefits of doing small changes instead of one big change:

• You get to keep your weekends.
• Any failures will be detected sooner within the deployment, allowing you opportunity to improve.
• Failures will not be as impactful.
• It’s easier to identify which change caused the issue, since there are less changes to begin with.

Create As-is Documentation

You’ll need to document the current state of the environment. This will help everyone move in the correct direction and avoid any misconceptions.

You can skip this step if you have documentation already.

Choose Code Version

You’ll need to determine which code version is best for your deployment. This includes 2 things:

• Find a version that has the newer features you want to take advantage of.
• Do a bug scrub. Review the vendor’s version notes and make sure any bugs on that version will not impact your organization.

For firewalls, I tend to stick with newer code to favor security and features. For routers and switches, I try to find the oldest, most stable code that has the features I need instead.

Configuration Prep

• Write any configuration templates where possible to save time.
• Make any final tweaks needed for each individual device.
  • Example: Switch access port configurations, unique site firewall rules and routing configurations.

Hardware Acquisition and Delivery

Self-explanatory, but sometimes this is the hardest part. Make sure your hardware refresh strategy accounts for supply chain issues and product availability!

Hardware Staging

The rest of these steps are repeated for each device that you are updating as part of your hardware refresh. These steps assume you’re going to ship it to wherever your network engineers are so they can stage them before sending them to wherever they will be installed. You’ll need to make some adjustments if they’re being shipped directly to the site.

Unbox and Apply Code Upgrades

Unbox the device and power it on. You’ll probably need basic connectivity, either to the internet or your FTP server depending on the type of device.

Ideally, you’d place it in a shelf where you have extra space in a rack, but being powered online at your desk works in a pinch.

Make sure you don’t throw away any of the original packaging. You’ll need it for later when we ship it to the site.

For routers and switches, we’ll just need to install the code version we chose earlier. It’s best to knock this out of the way before doing anything else. Configuration might be slightly different on older code versions, and it’s best to do the necessary reboots for software upgrades before it’s in production.

For devices like firewalls, you’ll also need to download any malware definition updates, URL lists, etc.

Apply configuration to device

Hopefully the templates you made earlier are pretty good and you can just copy/paste them in. If not, you’ll need to put some work in to make sure they’re applied correctly and working.

If your configuration templates are new, you may want to take some time testing/verifying your config before proceeding.

Create a Cable Diagram for Installation Instructions

You may want to make your instructions extra detailed here if the person doing the physical swap is less technical.

Box device with Cable Diagram

Take pictures as you put the device back into the box. This will come in handy with UPS/FedEx insurance claims if they happen to get damaged in shipping. It doesn’t happen often, but you’ll be glad you did it if they’re needed.

Make sure to include the cable diagram and any cables, SFP’s or accessories that might be needed on-site. I also include a console cable if the on-site contact does not have one already.

Ship Device to Site

I make sure to buy insurance with the courier. I also try to avoid transporting the devices myself, even if they can fit in my luggage/car to avoid any concerns or liability if the devices get damaged in transit.

Hardware Go-Live

You’ll finally get to realize the benefits of your hardware refresh once you’re done with this section of the hardware refresh strategy. In here, we’ll be coordinating with everyone involved and preparing for and implementing the change to bring the hardware online.

Establish On-Site Resource

Coordinate receiving the hardware and the cutover with whoever will be on-site.

Ideal situation: You have a fellow IT co-worker on-site who can help with the physical swap.

Reality: you have to work with the site-manager who may or may not be familiar with this equipment.

Create Change Plan

Review cutover plan, rollback plan, and test plan.

The idea is to do as much work as you can here, instead of during the change window. A lot of engineers are smart enough to fly by the seat of their pants.

If you’re not smart enough like me, or just don’t like the stress of having to think on the spot during a late change window, it’s a lot better to just do the engineering, design, and planning ahead of time. That way, during the change window, I’m just there to follow a checklist without needing to think too much.

r/networkingmemes – I have learned my lessons

Schedule and organize the maintenance window. Get CAB approval, buy-in from anyone needed, etc.

Review Change Plan with On-Site Resource

Meet with on-site contact to make sure they understand what’s needed from them during the maintenance window and give them an opportunity to ask any questions before-hand. Work with them to stage the device in the rack if there’s room.

Perform Cutover for Go-Live

How you do this is a whole subject in itself. The ideal cutover would be that both old and new devices are racked next to each other and you can just move the cables from the old one to the new one to complete the change.

Monitoring and Support

You’re almost done! Don’t forget to do all the little things needed once you complete your cutover.

It’s easy to skip these steps now that everyone is benefiting from the hardware refresh, but skipping over them in your strategy will make your job harder in the long run. Make sure you do these things so ensure you’ll be more successful supporting the hardware throughout it’s life cycle.

Monitoring

Add into your network monitoring system, syslog, etc. I try to do this immediately after finishing the cutover, or during the cutover as part of my test plan.

If you have any support agreements with vendors, you may need to update them as well.

First Day of Support

I try to set aside the first day of operations for fire watch. I avoid scheduling anything during this time so that I can either enjoy how good of a job I did, or fix any issues that come up as a result of the change. Hopefully your test plan is solid and took into account most scenarios and you have an easy day ahead of you.

Be sure to take note of any issues as you resolve them. Make adjustments to future deployments if necessary.

Even if your cutover is flawless, you want to make sure you have extra time to handle whatever comes up. For some reason, this is when people start using things they haven’t used in a while and they tell you your network change broke it, even if it had been broken for months prior.

Update Documentation

Documentation is critical to any hardware refresh strategy.

And no, this doesn’t count:

r/networkingmemes – Thaks for nothing

You’ll want to update your hardware inventory, along with any network diagrams, rack elevations, pictures, etc.

My inventory usually includes install location, model numbers and serial numbers. Even though you can grab them off the device or remotely, it helps to have them elsewhere in case you need them.

I’ve been in situations where I did not have access to the site myself and the device was no longer offline. I couldn’t start the RMA with the vendor until I got someone on-site to provide the serial number to me. After that, I just keep them all in a spreadsheet.

Bonus: How to Lead a Successful Hardware Refresh with e-Mayhem

What we do is difficult. We’re not worried that others will steal our process, because our secret ingredient is simply just doing a good job. If you want us to help make your hardware refresh successful, you can follow these steps below.

1. Contact our sales team to schedule a discovery call.
2. Discovery Call
   • e-Mayhem engineers will determine the current state of your network, and work with your team to determine what the end-state should look like.
   • e-Mayhem will help you choose and acquire the hardware, and design your network.
3. Sign contract for Value-Added hardware purchase.
4. Schedule a maintenance window with e-M.
   • Depending on your existing design, we may be able to do this without any visible user impact.
5. Physical Cutover – Can be done by e-M engineers for an additional cost + travel.
   • Receive already configured and upgraded equipment.
   • Follow included cable diagram to swap out hardware during maintenance window.
     • e-M’s engineers make a cable diagram specific for each device of what needs to be connected where and will be online to validate that this is done correctly and to perform testing to ensure a successful cutover.
6. Report any issues discovered during first day of support to e-M engineers on standby.

Thank you for checking this out. I hope you learned something new or enjoyed reading this. If you had any comments, questions, or just wanted to share your thoughts on this article, you can contact me at blog@e-mayhem.com

e-Mayhem helps companies successfully deliver business projects. We also help companies avoid losses associated with IT disruptions and security threats. You can learn more about our services at e-mayhem.com or by emailing sales@e-mayhem.com