How Often should Real-World DR plans be tested on production environments? I am getting blowback from my staff that the risks outweigh the benefits. I am looking to put these tests into practice while educating my staff that things can and will go wrong, which is why we do it on our timing, not when we are forced.

4k viewscircle icon9 Comments
Sort by:
CISO/CPO & Adjunct Law Professor in Finance (non-banking)2 years ago

I concur.

Test annually, and whenever you make a major change to your infrastructure or business processes. Most see the utility of testing when you go from physical to virtual servers or something similar, but changes in business process can be more material.  The DR plan should address recovery time objectives (RTOs), an estimate of how fast key systems can be restored. The order of restoration should be derived from the system’s importance to specific business processes, think welding robot software for an auto assembly line.  If your company is pivoting to make sweaters, then the welding robot software which previously was essential won’t be used any longer. The software supporting sweater production is now critical, so a revised DR plan and a new test is required.

As far as the person pushing back on testing - ask the naysayer if they are willing to accept responsibility (in writing) for the company’s inability to function after an incident.

Lightbulb on1
CIO in Manufacturing2 years ago

Test business critical environments annually. Assuming you have performed a BIA, you need to know that you can recover on the established RTOs and RPOs. 

Lightbulb on3 circle icon1 Reply
no title2 years ago

Thank you, this is where I am aiming to get to.

IT Manager in Construction2 years ago

Hello, in my view 1 time per year is enough.

Director in Manufacturing2 years ago

We do fully recovery of our SAP ERP once a year which also includes logins by our business people in HR, Finance, Accounting (Receivables and Payables). If it’s really a business critical application you better confirm you can get it functional again and confirm how long it takes

Lightbulb on3 circle icon1 Reply
no title2 years ago

Thank you for your input. Ideally, I would like to get a few tabletop exercises and VM test environment simulations done and a full DR in prod once a year.

Lightbulb on1
IT Manager in Construction2 years ago

For the production/live environment is a critical activity: you can choose between a full DR test or a test focused on the most crucial parts.
In my view, a DR should be focused on the worse scenario you can think about.

About the staff I see also here a dual approach: you can or you can't inform them during the test.
You will get insights on both cases but of course the Management must be informed early.

Lightbulb on1 circle icon2 Replies
no title2 years ago

Thank you for the comment. Yea I agree and it seems my train of thought is correct with the industry. Now to just get staff by in, focusing on one area at a time is a good start though thanks!

Lightbulb on1
no title2 years ago

Thanks to you Anthony!

Content you might like

Yes, we are GitOps mature18%

Yes, in some instances48%

No, we don't have a use-case for it19%

No, it's too expensive or cumbersome5%

No, and I'm unfamiliar with the term "GitOps"7%

View Results

We are currently using AI in our contact center17%

Our data is not easily/fully accessible between departments40%

We use conversational AI for web chats, but not for telephony25%

We just can't justify the investment15%

Other (please comment)1%

View Results