Incident Management & Communication
Incident Management & Communications
Procedures Guide Version 1.91
Last Updated:
December 2, 2009
Incident Management & Communication
This page left intentionally blank
2
Incident Management & Communication
Table of Contents Introduction................. ............................................................................................................. Introduction..................................................................... ......................................................... ............................... 4 Severity Level Definitions .................................................... ................................................................................................................ ............................................................ ...................... 5 Communication Checklist....................................... ........................................................ ................................................................................................ ........................................ 8 Manager On Call (MOC) .................................................... .......................................................................................................... ............................................................................ ...................... 8 IT Center .................................................. ........................................................ ....................................................................................................... ............................................... 9 Incident Manager On Call – IMOC List ............................................ ......................................................... .......... 10 Incident Manager On-Call (IMOC) ............................................. ....................................................... ................................................................. .......... 11 University IT Technical Staff / Technicians on Call .................................................. .................................. 12 University IT Director - of affected unit(s).................................................... unit(s)................................................................................................... ............................................... 12 Information Security ................................................. .................................................... .................................................................................... ................................ 13 CIO’s Office......................................................... ........................................................ ......................................................................................... ................................. 14 Communication Manager and/or Other Designated University IT Employees................................ ............ 15 Scribe ........................................................ ................................................................................................................ .................................................................................................... ............................................ 16 University IT Office Admins .................................................... .............................................................................................................. .......................................................... .......... 16 University IT Staff Members ................................................... ........................................................... .......... 16 Provost .................................................... ............................................................................................................ ...................................................................................................... .............................................. 16 President........................................ ........................................................ ............................................................................................................... ......................................................... 16 Other University Executives ..................................................... ............................................................................................................... .......................................................... .......... 17 Students................ ........................................................ ............................................................................................................ ................................................................................. ............................. 17 Faculty / Departments or Divisions ................................................... ................................................. .......... 17 University Staff................................................. Staff ................................................. ........................................................ ............................................................................................ .................................... 17 University Security ................................................. .................................................... ...................................................................................... .................................. 17 University Facilities ................................................ .................................................... ...................................................................................... .................................. 17 Rochester Management..................... Management......................................................................... ........................................................................................................... ......................................................... 17 University Legal.............................................. ................................................... ........................................... 17 University HR .................................................. ........................................................ ............................................................................................. ..................................... 17 Communication Call Log....................... ........................................................ .............................................................................................................. ........................................................ 18 Security Level Definitions .................................................... ................................................................................................................ ............................................................ .................... 20 Internal Communications Template......................... Template................................................................................ ............................................................................................. ...................................... 21 External Communications Template............................................... ............................................................ .......... 23 University IT Technician Form .................................................. ...................................................... .......................................................................... .................... 25 External Communication Matrix ........................................................ ................................................................................................................ .................................................................. .......... 26 Incident Command Center Wall Charts............................................ Charts ............................................ .......................................................... .......... 31 IT Alert (G2Alert) – Steps Steps to Send a Severity 3 IT Alert:.................. ................................................... ............... 40 ISD Manager On-Call - University IT (Data Center Services) Alert Notification ............................................... 41 Appendix......................................................... Appendix. ............................................................................................................... ...................................................................................................... ............................................... 43 Roles & Responsibilities................................... .................................................... ................................................................................................ ............................................ 44 Incident Manager On-Call (IMOC) ............................................. ....................................................... ................................................................. .......... 44 Manager On-Call (MOC)....................................... .................................................... ....................................................................................... ................................... 45 Communications Manager ................................................. ....................................................... ........................................................................... .................... 46 Web Content Hack – Immediate Actions ................................................ ........................................................ .......................................................... 51 Debrief Procedures................................................. ....................................................... ........................................................................................... .................................... 52 Debrief Agenda Template............................................................. Template.......... ................................................... .......................................................... .......... 53 Updating Procedures ........................................................ ............................................................................................................ ................................................................................. ............................. 54 Change Control ..................................................... ............................................................................................................ ............................................................................................ ..................................... 55 3
Incident Management & Communication
Introduction Leaders in the University Information Technology organization acknowledged the need to develop a wider view of incident management and communications. In the past, each University University IT department utilized its its own incident escalation path. Consistency in delivering incident management and expected communication levels were not meeting internal and external customer expectations, especially during high profile incidents. This Incident Management & Communication Procedures manual contains Severity 3 incident response tools. Severity 3 incidents are the highest level leve l and most critical of events that occur occu r within our organization. Immediate action is required by multiple people to assist in recovering services affected affected by the incident. By identifying scope and ownership of an incident early in the process, we can now triage to the appropriate teams, who in turn establish their communication protocols and management roles within the context of the broader incident management procedures. Incident management and communication processes that had been used independently across the organization are now merged into a single document and available across University IT. On-call escalation now has the ability to mobilize an Incident Manager On-Call (IMOC) who coordinates the Incident Command Center and communication methods to executives and customers. customers. Each department’s Manager On-Call (MOC) (MOC) can now concentrate on recovering services, without the the need to communicate with multiple people. people. Technicians will also benefit from these procedures by eliminating multiple communication paths and allowing them to concentrate on technical issues. Each Severity 3 incident will have a Communication Manager, assigned to assist with the creation of communication materials. A scribe will will detail incident events. After the recovery recovery from an incident, a mandatory debrief meeting will be scheduled to complete the the Sev 3. Documentation for the debrief methodology has been finalized and is included in this manual. A coordinated University IT response is essential to our business and services. Our customers demand it, our internal resources need it, and the Information Technology Services Incident Management & Communication Procedures Guide delivers it.
4
Incident Management & Communication
Severity Level Definitions Severity Level
3.0
2.5
2.0
1.5
1.0
0.0
Service Impact
Enterprise‐wide
Enterprise‐wide
Limited
Single or None
Single or None
Single or None
Immediate need for service
No immediate need for service
Single department affected by service interruption
Single user service impact
Single user service impact
Single user service impact
Scope may not be defined
Scope is defined
Aged General User Ticket
No Service Impact with complex elevated resolution
No Service Impact with elevated resolution
No Service Impact
Complete service outage
VIP User Ticket
Elevated User Ticket (Director, Manager)
General User Ticket
General User Ticket
General User Ticket
Triggers formal communication plan
Triggers formal communication plan
Triggers informal communication plan
Resolution by Tech Lead/System Lead
Resolution by Subject Matter Expert (SME)
Resolution by IT Center staff
IMOC/Director
Director
Manager
Level III: Tech Lead, System Lead
Level II: Subject Matter Expert
Call Agent
University IT Director
University IT Director
University IT Manager
Level III Support
Level II Support
Call Agent Only
University IT Manager
University IT Manager
Level III Support
Level II Support
Call Agent
IMOC
IMOC
CIO Office
Appropriate University IT Personnel
Senior Management
IT Center
Severity Level Defined
Multiple departments, groups, and individuals Enterprise‐wide impact University‐wide security violation/compromise
Decision Maker
Involvement
Call Agent
All University IT IT Center ISD (if Data (if Data Center or Network Related)
Who is Notified ‐ By Whom Immediate Notification
Additional Notifications
IMOC ‐ by Director
Director ‐ by Manager
Manager ‐ by Level III
Level III ‐ by Level II
Level II ‐ by Call Agent
Call Agent ‐ by User
CIO Office ‐ by IMOC
IMOC ‐ by Director
User ‐ by Ticket Assignee
User ‐ by Ticket Assignee
User ‐ by Ticket Assignee
User ‐ by Ticket Assignee
Senior Management ‐ by CIO Office
IT Center ‐ by Director
All University IT ‐ by Hyper‐Reach or Email
All University IT ‐ by Hyper‐Reach or Email
IT Center ‐ by Director ISD ‐ by Hyper‐Reach
Communication Plan Type
Formal*
Formal*
Informal
Informal
Informal
Informal
Communication Methods
Direct Contact (phone, in‐person)
Direct Contact (phone, in‐person)
Direct Contact (phone, in‐person)
Service Ticket
Service Ticket
Walk‐in, phone call, email, web form
Hyper‐Reach
Hyper‐Reach
Real‐time Communications
5
Incident Management & Communication ITENS
ITENS
6
Incident Management & Communication
7
Incident Management & Communication
Incident Inci dent Manager On Call Call – IMO IMOC C List Lis t Contact Operations (275-9194) or (275-1205) for most current IMOC list Group covers th e followi ng area(s) area(s):: The role of the Incident Manager Manager On Call is to lead Severity 3 and Severity 2.5 incidents. The Incident Manager On-Call is available 24x7.
Schedule ROTATION START DATE
PRIMARY
SECONDARY
TERTIARY
2009 May June July August September October November December
Crowley Wirley Barden Myers Fredericksen Crowley Wirley Barden
Wirley Barden Myers Fredericksen Crowley Wirley Barden Myers
Barden Myers Fredericksen Crowley Wirley Barden Myers Fredericksen
2010 January February March
Myers Fredericksen Crowley
Fredericksen Crowley Wirley
Crowley Wirley Barden
OTHER
Incident Management & Communication
Incident Inci dent Manager On Call Call – IMO IMOC C List Lis t Contact Operations (275-9194) or (275-1205) for most current IMOC list Group covers th e followi ng area(s) area(s):: The role of the Incident Manager Manager On Call is to lead Severity 3 and Severity 2.5 incidents. The Incident Manager On-Call is available 24x7.
Schedule ROTATION START DATE
PRIMARY
SECONDARY
TERTIARY
2009 May June July August September October November December
Crowley Wirley Barden Myers Fredericksen Crowley Wirley Barden
Wirley Barden Myers Fredericksen Crowley Wirley Barden Myers
Barden Myers Fredericksen Crowley Wirley Barden Myers Fredericksen
2010 January February March April
Myers Fredericksen Crowley Wirley
Fredericksen Crowley Wirley Barden
Crowley Wirley Barden Myers
OTHER
Personnel NAME Barden Crowley Fredericksen Myers Wirley
CALL CAL L FIRST 275.5458 275.8235 273.1714 273.1804 275.5615
CALL CAL L SECOND cell - 317.3398 cell - 733.1365 cell - 313.4003 cell - 208.0939 cell - 638.2591
OTHER AVAIL. AVA IL. home - 627.1602 pager - 220.3330 home - 586.5986 home - 349.7211 home - 671.9046
OTHER AVAIL . cottage - 315.536.6634 home - 924.3273
10
Incident Management & Communication Management Steps Communication Communication Flow Normal Business Hours (8:00am – 5:00pm)
4
Universit y IT Techn Techn ical Staff / Technic Technic ians on Call
Technicians will be required to be on-site unless otherwise directed by the IMOC or MOC. If MOC determines, technicians can forward internal calls for short periods of time. Troubleshoots problem and begins working on solutions.
Retrieve Technical Recovery Guides (TRG’s) for services affected. Provides regular updates to MOC. Participates in vendor calls as needed. Periodically checks in with other University IT staff members to assess the situation – be sure to include members in other locations. Avoid incoming customer calls. These are distractions to solving the issue at hand. If they are calling your phone, route them to the Call Centers (2). Do not speak with internal (Currents/Campus Times) or external (D&C/TV (D&C/TV stations) media. Direct them to University Communications.
After Hours
Technicians will be required to be on-site unless otherwise directed by the IMOC or MOC.
Troubleshoots problem and begins working on solutions. Retrieve Technical Recovery Guides (TRG’s) for services affected. Provides regular updates updates to MOC. MOC. If offsite, calls into MOC Phone Bridge if needed (1-866-603-2932 Access #6608484) Participates in vendor calls as needed. Periodically checks in with other University IT staff members to assess the situation – be sure to include members in other locations. Avoid incoming customers calls. These are distractions to solving the issue at hand. If they are calling your your phone, route them to the Call Centers (2). Do not speak with internal (Currents/Campus Times) or external
Incident Management & Communication Management Steps Communication Communication Flow Normal Business Hours (8:00am – 5:00pm)
4
Universit y IT Techn Techn ical Staff / Technic Technic ians on Call
Technicians will be required to be on-site unless otherwise directed by the IMOC or MOC. If MOC determines, technicians can forward internal calls for short periods of time. Troubleshoots problem and begins working on solutions.
5
Universit y IT Director - of affected unit(s)
Retrieve Technical Recovery Guides (TRG’s) for services affected. Provides regular updates to MOC. Participates in vendor calls as needed. Periodically checks in with other University IT staff members to assess the situation – be sure to include members in other locations. Avoid incoming customer calls. These are distractions to solving the issue at hand. If they are calling your phone, route them to the Call Centers (2). Do not speak with internal (Currents/Campus Times) or external (D&C/TV (D&C/TV stations) media. Direct them to University Communications.
Participates in discussions lead by MOC and IMOC.
Provides support to technical teams. Provides any other support that may be needed to help resolve the incident.
After Hours
Technicians will be required to be on-site unless otherwise directed by the IMOC or MOC.
Troubleshoots problem and begins working on solutions. Retrieve Technical Recovery Guides (TRG’s) for services affected. Provides regular updates updates to MOC. MOC. If offsite, calls into MOC Phone Bridge if needed (1-866-603-2932 Access #6608484) Participates in vendor calls as needed. Periodically checks in with other University IT staff members to assess the situation – be sure to include members in other locations. Avoid incoming customers calls. These are distractions to solving the issue at hand. If they are calling your your phone, route them to the Call Centers (2). Do not speak with internal (Currents/Campus Times) or external (D&C/TV stations) media. Direct them to University Communications.
May be onsite or working from home as determined by MOC. Participates in discussions lead by MOC. Provides support to technical teams. Provides any other support that may be needed to help resolve the incident.
12
Incident Management & Communication Management Steps Communication Communication Flow Normal Business Hours (8:00am – 5:00pm)
7
CIO’s Office
Receives details about incident from IMOC. Provides incident brief to Provost and President (12,13).
After Hours
Provides business perspective (big picture) for the incident.
Receives details about incident from IMOC. Decides if the Provost and President should be notified before the start of the next business day. Gathers with IMOC next business day morning to review event and provides business perspective (big picture) for the incident.
Incident Management & Communication Management Steps Communication Communication Flow Normal Business Hours (8:00am – 5:00pm)
7
CIO’s Office
Receives details about incident from IMOC. Provides incident brief to Provost and President (12,13).
After Hours
Provides business perspective (big picture) for the incident.
Receives details about incident from IMOC. Decides if the Provost and President should be notified before the start of the next business day. Gathers with IMOC next business day morning to review event and provides business perspective (big picture) for the incident.
14
Incident Management & Communication Management Steps Communication Communication Flow Normal Business Hours (8:00am – 5:00pm)
8
Communication Manager and/or Other Designated Universit y IT Employees
Gathers details about incident. Crafts messages for internal and external use. Identifies appropriate communication channels.
Deploys communications according to incident timeframe through identified channels/Working with MOC and IMOC. [All Channels]
Provides guidelines for communications to the Customer Service Centers and to the IT Admins so they can handle calls appropriately and deliver the same message (2,10).
Identifies channels for post-incident follow-up and helps prepare messages for those channels. Retain copy of all communications for debrief session and for audit purposes.
(Set up where main communication is taking place)
After Hours
Picks up the next business day to continue ongoing communications (internal and external) or to assist in closing out the incident. If incident is closed:
Sends final communications when incident is closed.
Identifies channels for post-incident followup and helps prepare messages for those channels. Retain copy of all communications for debrief session and for audit purposes. If incident is still open: Gathers details about incident and reviews CHRON.
Crafts messages for internal and external use. Identifies appropriate communication channels. Deploys communications according to incident timeframe through identified channels/Working with MOC and IMOC. [All Channels]
Incident Management & Communication Management Steps Communication Communication Flow Normal Business Hours (8:00am – 5:00pm)
8
Communication Manager and/or Other Designated Universit y IT Employees
Gathers details about incident. Crafts messages for internal and external use. Identifies appropriate communication channels.
Deploys communications according to incident timeframe through identified channels/Working with MOC and IMOC. [All Channels]
Provides guidelines for communications to the Customer Service Centers and to the IT Admins so they can handle calls appropriately and deliver the same message (2,10).
Identifies channels for post-incident follow-up and helps prepare messages for those channels. Retain copy of all communications for debrief session and for audit purposes.
(Set up where main communication is taking place)
After Hours
Picks up the next business day to continue ongoing communications (internal and external) or to assist in closing out the incident. If incident is closed:
Sends final communications when incident is closed.
Identifies channels for post-incident followup and helps prepare messages for those channels. Retain copy of all communications for debrief session and for audit purposes. If incident is still open: Gathers details about incident and reviews CHRON.
Crafts messages for internal and external use. Identifies appropriate communication channels. Deploys communications according to incident timeframe through identified channels/Working with MOC and IMOC. [All Channels] Provides guidelines for communications to the Customer Service Centers and to the IT Admins so they can handle calls appropriately and deliver the same message. Identifies channels for post-incident followup and helps prepare messages for those channels. Retain copy of all communications for debrief session and for audit purposes.
15
Incident Management & Communication Management Steps Communication Communication Flow Normal Business Hours (8:00am – 5:00pm)
9
Scribe (Set up where main communication is taking place)
Takes detailed notes during event to help complete the CHRON and serve as a record of the event. Types up info in CHRON template and distributes to team at regular intervals during incident. Prepares and send final CHRON at close of incident. Provides this info for debrief meeting.
After Hours
Picks up in the AM of next business day. If incident is closed:
Types up info in CHRON template and distributes to team at regular intervals during incident. Prepares and send final CHRON at close of incident. Provides this info for debrief meeting.
If incident is still open: Reviews CHRON already completed.
10
Universit y IT Office Admins
Uses guidelines for communications to customers when responding to calls that may come in from various areas.
Continues CHRON and takes detailed notes during the event. Types up info in CHRON template and distributes to team at regular intervals during incident. Prepares and send final CHRON at close of incident. Provides this info for debrief meeting.
In the AM of next business day: Uses guidelines for communications to customers when responding to calls that
Incident Management & Communication Management Steps Communication Communication Flow Normal Business Hours (8:00am – 5:00pm)
9
Scribe (Set up where main communication is taking place)
Takes detailed notes during event to help complete the CHRON and serve as a record of the event. Types up info in CHRON template and distributes to team at regular intervals during incident. Prepares and send final CHRON at close of incident. Provides this info for debrief meeting.
After Hours
Picks up in the AM of next business day. If incident is closed:
Types up info in CHRON template and distributes to team at regular intervals during incident. Prepares and send final CHRON at close of incident. Provides this info for debrief meeting.
If incident is still open: Reviews CHRON already completed.
10
Universit y IT Office Admins
Uses guidelines for communications to customers when responding to calls that may come in from various areas.
Universit y IT Staff Members
Uses guidelines for communications to customers when responding to calls that may come in from various areas.
Receives regular updates from CIO. Disseminates info as needed to key staff members.
11
12
Provost 13
President
Continues CHRON and takes detailed notes during the event. Types up info in CHRON template and distributes to team at regular intervals during incident. Prepares and send final CHRON at close of incident. Provides this info for debrief meeting.
In the AM of next business day: Uses guidelines for communications to customers when responding to calls that may come in from various areas.
In the AM of next business day: Uses guidelines for communications to customers when responding to calls that may come in from various areas.
Receives regular updates from CIO. Disseminates info as needed to key staff members.
16
Incident Management & Communication Management Steps Communication Communication Flow Normal Business Hours (8:00am – 5:00pm)
After Hours
14
Other University Executives 15
Students 16
Facul Facul ty / Departments or Divisions 17
University Staff 18
University Security 19
University Facilities 20
Rochester
Participates as required by incident.
Participates as required by incident.
Participates as required by incident.
Participates as required by incident.
Participates as required by incident, specifically when related to the Towne House building. 461-9440 or
Participates as required by incident, specifically when related to the Towne House
Incident Management & Communication Management Steps Communication Communication Flow Normal Business Hours (8:00am – 5:00pm)
After Hours
14
Other University Executives 15
Students 16
Facul Facul ty / Departments or Divisions 17
University Staff 18
University Security 19
University Facilities 20
Rochester Management 21
University Legal 22
Universit y HR
Participates as required by incident.
Participates as required by incident.
Participates as required by incident.
Participates as required by incident.
Participates as required by incident, specifically when related to the Towne House building. 461-9440 or 467-2442 after hours
Participates as required by incident, specifically when related to the Towne House building. 461-9440 or 467-2442 after hours
Participates as required by incident, specifically when security related.
Participates as required by incident, specifically when security related.
Participates as required by incident, specifically when security related.
Participates as required by incident, specifically when security related.
17
Incident Management & Communication
Communication Call Log Last revised On: 7/15/07 Who to contact
Notify? Yes/No
Contacted By
1st Contact At:
2nd Contact At:
3rd Contact At:
4th Contact At:
IT Center: Provide key facts so centers can handle incoming calls consistently and triage accordingly. IT Center @ 5-2000 Ops @ 5-9194 (TH Computer Room) University IT Incident Management: Provide key facts and begin IM team mobilization and communications. University IT Directors (Sev 3 VIP list) CIOs office @ 5-5240 Norm Acunis (for Email Sevs) Becky Kingcaid
Incident Management & Communication
Communication Call Log Last revised On: 7/15/07 Who to contact
Notify? Yes/No
Contacted By
1st Contact At:
2nd Contact At:
3rd Contact At:
4th Contact At:
IT Center: Provide key facts so centers can handle incoming calls consistently and triage accordingly. IT Center @ 5-2000 Ops @ 5-9194 (TH Computer Room) University IT Incident Management: Provide key facts and begin IM team mobilization and communications. University IT Directors (Sev 3 VIP list) CIOs office @ 5-5240 Norm Acunis (for Email Sevs) Becky Kingcaid (for Email Sevs or any Sev affecting Executives in Wallis) Information Security Office (as needed) Michelle Rogers Bill Waterhouse Main University IT Communication Channels: Provide high-level status of the event with updates as needed. 3-3999 Recording & Sev Page Sent University IT Notices Updated (University IT website) IT Center Plasma Screen University IT Org Phone Tree and/or G2 Alert University IT Office Admins: Provide key facts so this team can handle incoming calls consistently and provide departmental support as needed. CIO’s Office Finance/Admin/Comm Finance/Admin/Comm Office AA Office NC Office DC Office Security Office External to University IT: Provide high-level status of the event with updates as needed. Phonedown Netdown President’s Office @ 5-8356 Nicholas Bigelow @ 5-8549 (President of Faculty Senate) Provost’s Office @ 5-5931
18
Incident Management & Communication
Who to contact
All Campus Admins. (for email Sevs) ISD @ 5-3200 Highland Hospital Comm Ctr @ 473-2200 Michele Cairns @ 1-8463 Med Ctr Director’s office (Julie Choate, Roberta Parker) Comm Ctr @ 5-2222 (Voice Services including VM) College Dean’s Office @ 3-5000 University Security Office Highland Hospital Security University Facilities Office University Human Resources University Legal Students Faculty University Staff Members University IT Notices Post (ITENS) Campus Times Currents Digest (Email Daily) Currents (Print)
Notify? Yes/No
Contacted By
1st Contact At:
2nd Contact At:
3rd Contact At:
4th Contact At:
Incident Management & Communication
Security Level Definitions Department: Information Security – Guiding Criteria
Security Controlled (Sec. 3) Definition Information has the potential of being disclosed or altered that would: 1. Violate Laws, Regulations or Contractual Obligations 2. Significantly impact the reputation of the University OR
Examples Server has been compromised that has □ Student Social Security Numbers. Major worm outbreak is taking down □ email, HRMS, etc. Main University Web Page significantly □ defaced.
A significant and growing number of SERVICES are SERVICES are rendered unavailable without any operational remedy.
Security Related (Sec. 2) Definition Information has the potential of being disclosed or altered that would: 1. Cause Significant Harm to the University 2. Alter or disclose information regarding an individual or group in an unauthorized manner 3. Alter the results of Research or Business Processes in an unauthorized manner.
Examples Student Changes Grades. □ Researcher changes research data □ Worm outbreak is spreading rapidly □ across ResNet.
OR A significant and growing number of SYSTEMS are SYSTEMS are rendered unavailable without any operational remedy.
Security Notified (Sec. 1) Definition Information that has been deemed non-critical has the potential of being altered or disclosed, without adverse impact to the University OR A number of information systems are rendered unavailable without any operational remedy
Examples Known information is taken from a □ system without any impact. Individual systems are hit with a □ virus/worm. No trend across the University is detected.
Incident Management & Communication
Internal Communications Template Template ( I n t e r n a l – U n i v e r s i t y I T S t a f f O n l y )
Communications Contact:
Release Date:
Incident:
Communication Frequency:
University IT – Internal Audience: (check all that apply)
Who needs the information?
1 Time Only
Initial Comm + Multiple Updates
University IT – ALL Employees ----------------- CIO Directors University IT Managers University IT Office Admins University IT Operations Centers (IT Center/NCS
Ops/DCS Ops)
Executive Support Team University IT Student Workers (IT Center) N&C EC A&A S&P Computer Sales/Store University IT Finance & Admin Other
What information do they need?
Key Facts:
•
Item 1
•
Item 2
•
Item 3
•
Item 4
•
Item 5
•
Item 6
Channels: (check all that apply)
Email Web Phone/Conf. Bridge ITENS/University IT Home Page G2 Alert ext. 3-3999 In Person/Meeting
Other
University IT Hotline – for follow-up/summary
What’s the best way to reach them?
Incident Management & Communication
Initial Communication Copy
Version 1:
Version 2:
Version 3:
Special Instructions/Notes:
Communication Channel
University IT Audience
Assigned To
Copy Version
Updates Time
Date
Message
Page 22
Channels
Last Revised On: 4/17/06
Incident Management & Communication
External Communications Template Template (External – University Community and Press) Communications Contact:
Release Date:
Incident:
Communication Frequency:
1 Time Only
Initial Communication + Additional Updates as Needed
Entire University
External Community Audience: - - - - - - - - - - - - - - - - - -
(check all that apply)
Who needs the information?
All Faculty (All Schools) Staff All Students (All Schools) Student Workers (University IT) Residential Assistants (RAs) University Administration Department Administrators Deans (All Schools) Provost President VP of Communications Medical Center/ISD Medical Center/Staff Medical Center/ Communications Center Highland Hospital Communications Center Memorial Art Gallery Telephone Directory Contacts Key University IT Contacts University Legal University Security Office University Facilities University Human Resources Dept. Campus Times/Currents Local Press/TV and Print Other (Use this area for communications to specific Colleges)
Channels: (check all that apply)
What’s the best way to reach them?
Email Web Phone Currents Digest ITENS - University IT Home Page G2 Alert Fax In Person – Visit various locations IT Center Plasma Screen Flyers – post in relevant areas Other
Following Incident: Currents Print Campus Times Flyer/Postcard Follow-up Phone Call
Incident Management & Communication
What information do they need?
Key Facts:
•
Item 1
•
Item 2
•
Item 3
•
Item 4
•
Item 5
Communication Copy
Version 1:
Version 2:
Version 3:
Special Instructions/Notes:
Communication Channel
Audience (External to University IT)
Assigned To
Copy Version
Updates Time
Date
Message
Channel(s)
Incident Management & Communication
University IT Technician Form: Incident Management & Communications Time Alert ed Date Time
Alert ed by
Notif ied OPS OPS 275-9194 275-9195 220-3283 pager
Message Message of init ial alert:
Time OPS OPS Notifi ed
Vendor Vendor Case / Contact:
Systems affected
Servi Servi ces affected
MOC MOC Conf Conf erence Call Call B rid ge & Pin 1-866-603-2932 pin pin 6608484#
IMOC IMOC Conference Call Br idg e & Pin 1-866-871-2663 1-866-871-2663 or 273-3311 273-3311 pin 144357#
MOC
Time
IMOC IMOC
Event
SysAd min (s)
MOC MOC Notificatio n & Updates
Incident Management & Communication
External Communication Matrix
External To University IT
Who To Contact
Who Can Contact (from Universi ty IT)
Web
Email
Phone
College o of A Science, a and E Engineering Arts, S Deans Vice Provost and Dean of the College Faculty
Peter Lennie
[email protected]
3-5000
Dean of The College
Richard Feldman
[email protected]
3-5001
Dean of the School of Engineering and Applied Sciences
Robert Clark
[email protected]
5-4151
Vice Provost and Dean of Research and Graduate Studies
Wendi Heinzelman
[email protected]
5-4153
Dean of Sophomores Sophomores
Vicki Roth
[email protected] [email protected]
5-9049
Dean of Freshmen
Marcy Kraus
[email protected] [email protected]
5-2354
School of Engineering Computing and Networking Group (CNG)
John Simonson John Strong Jim Prescott Bob Lindholm
5-3106 5-4873 5-8265 5-0870
[email protected]
Department Heads All Faculty All Students Eastman S School o of M Music Dean/Director
Doug Lowry
263-2807
Computing Services
Jeremy Beyette
[email protected] [email protected]
David Guzick
[email protected] [email protected]
[email protected]
4-1160
School o of M & D Medicine & Dentistry Dean
5-0017
26
Incident Management & Communication
External Communication Matrix
External To University IT
Who To Contact
Who Can Contact (from Universi ty IT)
Email
Web
Phone
School o of N Nursing Dean
Kathy Parker
[email protected]
5-8902
[email protected] [email protected]
5-3316
[email protected] [email protected] r.edu
3-5215
[email protected]
5-8300
William E E. S Simon G Graduate S School o of B Administration Business A Dean
Mark Zupan
Department of IT
Joe Scacchetti
Margaret W Warner G Graduate S School o of E and H Human D Development Education a Dean Warner School Information Technology Service
Raffaella Borasi Dave Garcia
R iver C Campus L Libraries Dean
Susan Gibbons
[email protected]
5-4461
Information Technologies
Mike Bell
[email protected] [email protected]
5-6875
Medical C Center/Strong H Health/Highland Information Systems Division (ISD) Communications Center (Strong)
Jerry Powell
[email protected] [email protected] u
784-6118
Incident Management & Communication
External Communication Matrix
External To University IT
Who To Contact
Who Can Contact (from Universi ty IT)
Web
Email
Phone
School o of N Nursing Dean
Kathy Parker
[email protected]
5-8902
[email protected] [email protected]
5-3316
[email protected] [email protected] r.edu
3-5215
[email protected]
5-8300
William E E. S Simon G Graduate S School o of B Administration Business A Dean
Mark Zupan
Department of IT
Joe Scacchetti
Margaret W Warner G Graduate S School o of E and H Human D Development Education a Dean Warner School Information Technology Service
Raffaella Borasi Dave Garcia
R iver C Campus L Libraries Dean
Susan Gibbons
[email protected]
5-4461
Information Technologies
Mike Bell
[email protected] [email protected]
5-6875
Medical C Center/Strong H Health/Highland Information Systems Division (ISD)
Jerry Powell
784-6118
[email protected] [email protected] u
Communications Center (Strong) Communications Center (Highland) Security (Strong) Security (Highland) Facilities (Highland)
27
Incident Management & Communication
External Communication Matrix
External To University IT
Who To Contact
Who Can Contact (from Universi ty IT)
Email
Web
Phone
University A Administration President
Joel Seligman
[email protected] [email protected]
6-3262
Melinda Smith
[email protected] [email protected]
5-5931
Assistant Provost
Kathleen Moore
[email protected] [email protected]
5-2497
VP & General Secretary, Senior Advisor to the President, and University Dean
Paul J. Burgett
[email protected] [email protected]
3-2284
VP of Communications
William Murphy
Deputy to the President Provost Provost Exec Assistant
Communications Administrator Sr. VP of Finance & Administration/CFO Admin. Asst.
Lamar Murphy Ralph Kuncl
5-4124
Maureen Baisch
[email protected] [email protected]
5-4127
Ronald J. Paprocki
[email protected] [email protected]
5-2800
Helen W. Kostizak
[email protected] [email protected]
5-2792
Incident Management & Communication
External Communication Matrix
External To University IT
Who To Contact
Who Can Contact (from Universi ty IT)
Web
Email
Phone
University A Administration President
Joel Seligman
[email protected] [email protected]
6-3262
Melinda Smith
[email protected] [email protected]
5-5931
Assistant Provost
Kathleen Moore
[email protected] [email protected]
5-2497
VP & General Secretary, Senior Advisor to the President, and University Dean
Paul J. Burgett
[email protected] [email protected]
3-2284
VP of Communications
William Murphy
Deputy to the President Provost Provost Exec Assistant
Communications Administrator Sr. VP of Finance & Administration/CFO Admin. Asst. Sr. VP for Institutional Resources Secretary
Lamar Murphy Ralph Kuncl
5-4124
Maureen Baisch
[email protected] [email protected]
5-4127
Ronald J. Paprocki
[email protected] [email protected]
5-2800
Helen W. Kostizak
[email protected] [email protected]
5-2792
Douglas W. Phillips
[email protected] [email protected]
5-3311
Dianne Wittman
[email protected] [email protected]
5-8051
Sr. VP & Chief Advancement Officer
James D. Thompson
[email protected] jim.thompson@rocheste r.edu
3-2158
Sr. VP & Vice Provost for Health Affairs and Medical Center CEO
Brad Berk
[email protected] [email protected]
5-3407
VP and General Counsel
Sue S. Stewart
[email protected] [email protected]
3-5824
28
Incident Management & Communication
External Communication Matrix
External To University IT
Who To Contact
Who Can Contact (from Universi ty IT)
Email
Web
Phone
Memorial A Art G Gallery Grant Holcomb
[email protected] [email protected]
6-8902
Director
Robert McCrory
[email protected] [email protected]
5-4973
LLE Computer Support
Alex Rysken
[email protected] [email protected]
5-5333
The Mary W. and Donald R. Clark Director Laboratory f f or L Laser E Energetics
Other U University D Departments Security Office 3-4567
Facilities Human Resources
[email protected] [email protected]
5-8747
Office of Communications Public Information Coordinator
Sharon Dickman
[email protected] [email protected] du
5-4128
Publicist
Helene Snihur
[email protected] [email protected]. edu
5-7800
Incident Management & Communication
External Communication Matrix
External To University IT
Who To Contact
Who Can Contact (from Universi ty IT)
Email
Web
Phone
Memorial A Art G Gallery Grant Holcomb
[email protected] [email protected]
6-8902
Director
Robert McCrory
[email protected] [email protected]
5-4973
LLE Computer Support
Alex Rysken
[email protected] [email protected]
5-5333
The Mary W. and Donald R. Clark Director Laboratory f f or L Laser E Energetics
Other U University D Departments Security Office 3-4567
Facilities Human Resources
[email protected] [email protected]
5-8747
Office of Communications Public Information Coordinator
Sharon Dickman
[email protected] [email protected] du
5-4128
Publicist
Helene Snihur
[email protected] [email protected]. edu
5-7800
Editor, Currents
Jenny Leonard
[email protected] [email protected]. edu
5-6076
Web Editor
Lori Packer
[email protected]
5-5277
Other Telephone Directory Contacts Key University IT Contacts Residential Assistants University IT Student Workers
29
Incident Management & Communication
External Communication Matrix
External To University IT
Who To Contact
Who Can Contact (from Universi ty IT)
Email
Web
Phone
University Health Services (Director) Cary Jensen
[email protected] [email protected]
5-8928
Susan B. Anthony Center for Women's Leadership
Nora Bredes
[email protected] [email protected]
5-9283
University Intercessors
Gerald Gladstein Frederick Jefferson Ruth Lawrence Kathy Sweetland
International Services Office (Director) Office of Technology Transfer
Media ( (Internal t to U U o of R External) R a and E Campus Times
Various
[email protected]
5-5342
Currents Digest
Jenny Leonard
[email protected] [email protected]
5-6076
Currents (Print)
Jenny Leonard
[email protected] [email protected]
5-6076
Incident Management & Communication
External Communication Matrix
External To University IT
Who To Contact
Who Can Contact (from Universi ty IT)
Web
Email
Phone
University Health Services (Director) Cary Jensen
[email protected] [email protected]
5-8928
Susan B. Anthony Center for Women's Leadership
Nora Bredes
[email protected] [email protected]
5-9283
University Intercessors
Gerald Gladstein Frederick Jefferson Ruth Lawrence Kathy Sweetland
International Services Office (Director) Office of Technology Transfer
Media ( (Internal t to U U o of R External) R a and E Various
Campus Times
5-5342
[email protected]
Currents Digest
Jenny Leonard
[email protected] [email protected]
5-6076
Currents (Print)
Jenny Leonard
[email protected] [email protected]
5-6076
Local TV Stations
Sharon Dickman
[email protected] [email protected]
5-4128
Local Newspapers Newspapers
Sharon Dickman
[email protected] [email protected]
5-4128
Local Radio Stations
Sharon Dickman
[email protected] [email protected]
5-4128
30
Incident Management & Communication
Incident Command Center C enter Wall Wall Charts
Respond Time
Action Are Employees Safe? x13
Injured:
Contact Security (if necessary) x13 Contact Facilities (if necessary) x3-4567
Security Contact:
Contact Rochester Management (if necessary) University IT Security Controlled Event?
Personnel On-Site:
Contact Information:
SECURITY CONTROLLED EVENT if either of the following exist:
3.
Information has the potential of being disclosed or altered that would:
Incident Management & Communication
Incident Command Center C enter Wall Wall Charts
Respond Time
Action Are Employees Safe? x13
Injured:
Contact Security (if necessary) x13 Contact Facilities (if necessary) x3-4567
Security Contact: Personnel On-Site:
Contact Rochester Management (if necessary) University IT Security Controlled Event?
3.
{ Contact Information}
4.
Contact Information:
SECURITY CONTROLLED EVENT if either of the following exist:
Information has the potential of being disclosed or altered that would: a. Violate Laws, Regulations or Contractual Obligations b. Significantly Impact the University’s Reputation OR A significant and growing number of SERVICES are rendered unavailable without any operational remedy .
Contact University IT Security immediately University IT SECURITY WILL COORDINATE RECOVERY ACTIVITIES/COMMUNICATIONS
31
Incident Management & Communication
Time
Action University IT Security Controlled Event?
SECURITY RELATED EVENT if either of the following exist:
1.
{ Contact Information} 2.
Information has the potential of being disclosed or altered that would: 4. Cause Significant Harm to the University 5. Alter or disclose information regarding an individual or group in an unauthorized manner 6. Alter the results of Research or Business Processes in an unauthorized manner. OR A significant and growing number of SYSTEMS are rendered unavailable without any operational remedy .
Contact University IT Security IMOC will engage University IT Security to assist in recovery SECURITY NOTIFICATION EVENT if either of the following exist:
1.
2.
Information that has been deemed non-critical has the potential of being altered or disclosed, without adverse impact to the University OR A number of information systems are rendered unavailable without any operational remedy
CONTACT University IT SECURITY – NOTIFICATION ONLY
Severity 3 Declared
Declared By:
Incident Management & Communication
Time
Action University IT Security Controlled Event?
SECURITY RELATED EVENT if either of the following exist:
1.
{ Contact Information} 2.
Information has the potential of being disclosed or altered that would: 4. Cause Significant Harm to the University 5. Alter or disclose information regarding an individual or group in an unauthorized manner 6. Alter the results of Research or Business Processes in an unauthorized manner. OR A significant and growing number of SYSTEMS are rendered unavailable without any operational remedy .
Contact University IT Security IMOC will engage University IT Security to assist in recovery SECURITY NOTIFICATION EVENT if either of the following exist:
1.
2.
Information that has been deemed non-critical has the potential of being altered or disclosed, without adverse impact to the University OR A number of information systems are rendered unavailable without any operational remedy
CONTACT University IT SECURITY – NOTIFICATION ONLY
Severity 3 Declared
Declared By:
Incident Command Center Contact Information
Phone Numbers: Fax Numbers:
Help Desk Notifications CIO Notification
Who
IT Center
x5-2000 st
Contact Dave Lewis – Must Make Verbal Contact; Cell 1 , Home Phone 2
nd
32
Incident Management & Communication
Control Time
Action Technicians On-Site? ISD Comm Bridge Setup (if necessary) 1-866-945-2255 Access Code: 608965#
IMOC Comm Bridge Setup
Notate Time Sent in “Command Center Information”
x33311 or 1-866-871-2663 144357#
MOC Comm Bridge Setup
Notate Time Sent in “Command Center Information”
1-866-609-2932 6608484
IT Alert Sent
Notate Time Sent in “Communication Updates”
www.g2alert.com Notate Time Sent in “Communication Updates”
University IT-ORG Email Sent Notate Time Sent in “Communication Updates”
University IT Notices Updated
Incident Management & Communication
Control Time
Action Technicians On-Site? ISD Comm Bridge Setup (if necessary) 1-866-945-2255 Access Code: 608965#
IMOC Comm Bridge Setup
Notate Time Sent in “Command Center Information”
x33311 or 1-866-871-2663 144357#
MOC Comm Bridge Setup
Notate Time Sent in “Command Center Information”
1-866-609-2932 6608484
IT Alert Sent
Notate Time Sent in “Communication Updates”
www.g2alert.com Notate Time Sent in “Communication Updates”
University IT-ORG Email Sent Notate Time Sent in “Communication Updates”
University IT Notices Updated Notate Time Sent in “Communication Updates”
x3-3999 NCS Notification Customer Communications
33
Incident Management & Communication
Incident #1 Details Brief Description of Problem
Services & Servers Affected
Customer(s) Impacted
Resource Assigned
Current Status
Relief Person & Next Shift
Incident Management & Communication
Incident #1 Details Brief Description of Problem
Services & Servers Affected
Customer(s) Impacted
Resource Assigned
Current Status
Relief Person & Next Shift
34
Incident Management & Communication
Incident #2 Details Brief Description of Problem
Services & Servers Affected
Customer(s) Impacted
Resource Assigned
Current Status
Relief Person & Next Shift
Incident Management & Communication
Incident #2 Details Brief Description of Problem
Services & Servers Affected
Customer(s) Impacted
Resource Assigned
Current Status
Relief Person & Next Shift
35
Incident Management & Communication
Command Center Information Address:
Fax #:
Location
IMOC IMOC Communication Conference Bridge Only MOC Conference Bridge ISD Conference Bridge
MOC/Technician Communication Only If necessary
Phone #
Access #/Pin Code
Phone #
Access #/Pin Code
Phone #
Access #/Pin Code
1-866-945-8855
608965
Incident Management & Communication
Command Center Information Address:
Fax #:
Location
IMOC IMOC Communication Conference Bridge Only MOC Conference Bridge ISD Conference Bridge
MOC/Technician Communication Only If necessary
Phone #
Access #/Pin Code
Phone #
Access #/Pin Code
Phone #
Access #/Pin Code
1-866-945-8855
608965
36
Incident Management & Communication
Personnel IMOC IMOC Communication Assistant
Scribe Communication Manager
MOC – AA MOC – DC
Name
Contact Info
Location
Relief Person & Next Shift
Incident Management & Communication
Personnel
Name
Contact Info
Location
Relief Person & Next Shift
IMOC IMOC Communication Assistant
Scribe Communication Manager
MOC – AA MOC – DC MOC – ISO MOC – NC
37
Incident Management & Communication
Communication Updates Vehicle
Contact Info
IT Center
275-2000
Data Center Operators
275-9194 275-1205
IT Alert
https://g2alert.com
University IT Organization Updated
1. University IT-ORG email list (if avail) 2. IT Alert 3. Phone Tree
University IT Notices Updated Phone Update
273-3999
Performed By
Last Update
Incident Management & Communication
Communication Updates Vehicle
Contact Info
IT Center
275-2000
Data Center Operators
275-9194 275-1205
IT Alert
https://g2alert.com
University IT Organization Updated
Performed By
Last Update
1. University IT-ORG email list (if avail) 2. IT Alert 3. Phone Tree
University IT Notices Updated Phone Update
273-3999
38
Incident Management & Communication
University IT Contact
Service/ Server
Vendor Contact Information Company Contact Name
Phone #
Case #
Incident Management & Communication
University IT Contact
Service/ Server
Vendor Contact Information Company Contact Name
Phone #
Case #
39
Incident Management & Communication
IT Alert (G2Alert) – Steps to Send a Severity 3 IT Alert: 1. 2. 3. 4.
Gather information concerning concerning the incident: incident: Incident details, details, Service(s) Application(s) and and Server(s) affected https://www.g2alert.net & https://www.g2alert.net & login Choose Messages, then choose Send A Message Choose “Create or Edit a Message” or select an existing Message from the pulldown list
If Creating a New Message Message Setup
Time of Day Business Hours
After Hours
Select Message
Choose “Create A Message”
Create A Message
Choose “Start with a Template”, and choose appropriate timeframe template Business Hours Hours - Template Template After Hours Hours - Template Template
Remember: You must fill in EACH method below for the message message to reach recipients recipients via that method method (Text, SMS and Voice) Voice) Message Name
***** Change the Message Name *****
Voice Messages Email Messages
Change {service/application/server/event {service/application/server/event} }
Change {service/application/server/event {service/application/server/event} }
Incident Management & Communication
IT Alert (G2Alert) – Steps to Send a Severity 3 IT Alert: 1. 2. 3. 4.
Gather information concerning concerning the incident: incident: Incident details, details, Service(s) Application(s) and and Server(s) affected https://www.g2alert.net & https://www.g2alert.net & login Choose Messages, then choose Send A Message Choose “Create or Edit a Message” or select an existing Message from the pulldown list
If Creating a New Message Message Setup
Time of Day Business Hours
After Hours
Select Message
Choose “Create A Message”
Create A Message
Choose “Start with a Template”, and choose appropriate timeframe template Business Hours Hours - Template Template After Hours Hours - Template Template
Remember: You must fill in EACH method below for the message message to reach recipients recipients via that method method (Text, SMS and Voice) Voice)
***** Change the Message Name *****
Message Name Voice Messages Email Messages Type as you would say it; you may need spaces between letters letters
Change {service/application/server/event {service/application/server/event} } to reflect actual incident.
Change {service/application/server/event {service/application/server/event} } to reflect actual incident.
Press Send – go to Sending Message below
Press Send – go to Sending Message below
Text Messages Maximum 108 characters characters Send
If Editing a Existing Message Message Setup Select Message
Business Hours
Time of Day Choose Edit or Copy
After Hours
Remember: You must fill in EACH method below for the message message to reach recipients recipients via that method method (Text, SMS and Voice) Voice) Voice/Email/Text Messages
Change message to reflect actual incident.
Change message to reflect actual incident.
Send
Press Send – go to Sending Message below
Press Send – go to Sending Message below
Sending Message Message Setup
Time of Day Business Hours
After Hours
List
(CL) ALERT: UnivIT Only DIRs/MGRs (CL) ALERT: ISD Only MOCS & Bat Line
Request Confirmation Of a Receipt
Always Choose “Yes”
Prompt Voice Message Recipients to Join A Conference Call
No
Yes
Device Preferences
Choose “Send to ALL”
Choose “Send to Preferred Only”
Send Alert
Always choose “Now”
5. Press “Continue” in the lower right hand corner. distribute the message. 6. After verifying the Send Message, choose “Send”. This will invoke the service to distribute Version 1.9 Confidential
40
Incident Management & Communication
ISD ISD Manager Manager On-Call On-Call - Univ ersity ersi ty IT (Da (Data ta Center Center Servi ces) Alert Notif icati on Universit y Data Center Center Servic Servic es uses IT Alert, Alert, automatic notific ation softw are, that contacts specified individuals automatically, via cell phone, pager, home ph one, e-mail, fax, or other, in the event of an emergency.
The ISD Manager Manager on Call will be contacted by IT Alert for any Severity 3 incident. At all hours, IT Alert will contact the ISD Manager on Call listed below. NOTE: The ISD Manager on Call will follow the ISD Incident Management Procedures to activate and contact ISD Management as applicable. Single system outages will be escalate escalated d thr ough normal University IT escalation escalation procedures procedures.. IT Alert will not be activated. The ISD Manager on Call will be contacted by each of their communication devices. Contact will be made in the order shown below, pager, cell phone, work phone, home phone, and e• mail. The pecking order will continue until all of your devices have been reached. • The IT Alert Alert Notification c ontacts all devices; it does not stop if it reaches reaches you by one of y our • contact devices, even even if you h ave confirmed receipt. receipt.
ISD Manager On Call Schedule 2009 (see next page)
Version 1.9 Confidential
41
Incident Management & Communication
ISD Manager On Call Schedule 2009
Section 2. 2. Contact Contact and Commu Commu nication Information Primary Secondary OnCall
Business Phone
Pager Or Cell Phone
Home Phone
Start Time
End Time
Mgr Person OnCall
6/16/09
7/6/09
Rick Haverty
Primary
784‐6126
313‐0485
586‐6384
6/16/09
7/6/09
Dave Lindsey
Secondary
784‐2949
314‐5665
315‐589‐8776
7/7/09
7/20/09
Dave Lindsey
Primary
784‐2949
314‐5665
315‐589‐8776
7/7/09
7/20/09
Diane Koretz
Secondary
341‐0403
734‐8976
315‐524‐7430
7/21/09
8/3/09
Chip Nimick
Primary
415‐9053
671‐7570
7/21/09
8/3/09
Gary Scialdone
Secondary
350‐9588
787‐1639
8/4/09
8/17/09
Gary Scialdone
Primary
784‐6115 784‐2480 /275‐1120 784‐ 2480/275‐ 1120
350‐9588
787‐1639
8/4/09
8/17/09
Nancy Bales
Secondary
784‐8322
507‐6791
393‐1229
8/18/09
8/31/09
Nancy Bales
Primary
784‐8322
393‐1229
8/18/09
8/31/09
Sue Graves
Secondary
784‐2435
9/1/09
9/14/09
Sue Graves
Primary
784‐2435
507‐6791 730‐2299/755‐ 5395cell 730‐2299/755‐ 5395cell
9/1/09
9/14/09
Ted Vaczy
Secondary
784‐6002
576‐3651
624‐2792
9/15/09
10/5/09
Ted Vaczy
Primary
784‐6002
576‐3651
624‐2792
9/15/09
10/5/09
Chip Nimick
Secondary
784‐6115
415‐9053
671‐7570
10/6/09
10/19/09
Diane Koretz
Primary
341‐0403
734‐8976
315‐524‐7430
10/6/09
10/19/09
Kathrin Kenny
Secondary
784‐6121
474‐3569
315‐524‐4821
10/20/09
11/2/09
Kathrin Kenny
Primary
784‐6121
474‐3569
315‐524‐4821
10/20/09
11/2/09
Tina DePalo
Secondary
784‐8338
507‐9270
507‐9270
11/3/09
11/16/09
Tina DePalo
Primary
784‐8338
507‐9270
11/3/09
11/16/09
Halle McNaney
Secondary
784‐4275
245‐1884/880‐1022
11/17/09
11/30/09
Halle McNaney
Primary
784‐8275
507‐9270 245‐1884/880‐ 1022 245‐1884/880‐ 1022
11/17/09
11/30/09
Tina DePalo
Secondary
784‐8338
507‐9270
507‐9270
12/1/09
12/14/09
Tina DePalo
Primary
784‐8338
507‐9270
507‐9270
12/1/09
12/14/09
Marty Bush
Secondary
784‐8331
472‐4184
458‐3519
12/15/09
1/3/10
Marty Bush
Primary
784‐8331
472‐4184
458‐3519
12/15/09
1/3/10
Dawn Robinson
Secondary
784‐6159
820‐9274
383‐1213
Version 1.9 Confidential
335‐3276 335‐3276
245‐1884/880‐1022
42
Incident Management & Communication
Appendix
Version 1.9 Confidential
43
Incident Management & Communication
Roles & Responsibilities Responsibilities Incident Manager On-Call (IMOC) The Incident Manager On-Call is a Director-level Director-level role and is responsible for managing University University IT-wide incidents. The IMOC serves as a liaison to University executive offices and the University IT Managers On-Call during SEVERITY 3 incidents (defined below). They are on-call for one month, and are supported by a secondary and tertiary tertiary backup. The IMOC is available 24x7 during their monthly assignment. Definition Definition of Severity 3: The problem has a critical impact on key functions within the University University or its reputation. Resolution takes highest precedence.
IMOC responsibilities: Evaluate the situation and gathers al l the facts from all Managers On-Call. Determine if the MOCs should be onsite during an incident that occurs outside normal business hours (8am-5pm weekdays), also known as “AFTER HOURS”. Oversee the Severity 3: Communication Checklist & Call L og process Contact the CIO Work directly with MOCs & technical teams as necessary Notifies University IT Information Security Office to review incident and determine if a security breach has occurred. Serve as incident Communications Manager and oversee the gathering of information (CHRON) and customer communications. Determines the need/location of an Incident Command Center to manage the incident (also referred to as the “University IT War Room”) Designate an incident scribe. In direct contact with with the incident scribe and oversees oversees all notifications to University IT ORG and if necessary, key University division contacts; contacts; President’s Office, Office, Provost’s Office, Office of Communications, College Dean’s Office, URMC (School of Nursing), Simon School, Warner School and Eastman School of Music.
IMOC IMOC Schedule Changes If an IMOC is unavailable (sick, vacation, etc.), the IMOC is responsible for the following: 1. Notifying the secondary or tertiary IMOC IMOC to serve in their place 2. Notify University University IT Production Control of the change in schedule a. Use the “ITS Production Control” distribution distribution list in the GAL b. Include start and stop dates and times for schedule modification University IT Production Control will provide the IMOC update to the following: 1. SharePoint On-Call List https://sharepoint.its.rochester. https://sharepoint.its.rochester.edu/sites/DataCenter_OnC edu/sites/DataCenter_OnCall/default.aspx all/default.aspx 2. University IT Directors DL “IT Leadership” in the GAL 3. University IT Managers “ITS Managers” in the GAL 4. IT Centers
[email protected] and/or
[email protected] and/or 5-2000 5. Ida Gatto
[email protected] and/or
[email protected] and/or 5-9510 Version 1.9 Confidential
44
Incident Management & Communication
Manager On-Call (MOC) The Manager On-Call is a Manager-level Manager-level role and is responsible for managing business unit level incidents. The MOC serves as a liaison for after hours notifications of the situations that are subject to off-hours resolution; receive calls from the after hours dispatch dispatch service, provide severity level review, review, triage/filter and dispatch dispatch staff as required. They are generally on-call for one week, and are supported by backup MOCs. MOCs. The MOC is available 24x7 during their their assignment. For severe service outages referred to as SEVERITY 3, the MOC is required to contact the Incident Manager On-Call (IMOC). Definition Definition of Severity 3: The problem has a critical impact on key functions within the University University or its reputation. Resolution takes highest precedence. MOC is responsible to: Ensure that each call is reported [Chronology, [ Chronology, HEAT or some other logging tool?] tool?] Only summary information needs to be recorded for all of the single user problems. Severity 2 and Severity 3 problems require communication as specified to ensure proper notification of service outages and also require logging basic chronology of events to report significant progress in solving problems. General Rule - State what you can do for the customer and not what you can’t do by positive negotiations. Offer your office phone number to the IT Center and the Operations Center number for inquiries by the customer on the next business day. Update the University IT MOC list, and individual unit on-call schedules should be used to determine the appropriate triage and notification(s) Certain service disruptions require contact with general dispatch points:
ISD Help Desk at x53200 can x53200 can be your reference point for any ISD staff on call for desktop or Med Center department network issues, such as with Omega. If x5-3200 is closed (after hours), you will be rolled over to the Data Center (x5-9194 or x5-9195).
Energy Management at x34567, x34567, a.k.a. Customer Service Center, a.k.a. Energy Operations Center, is your link to all trades-people in Facilities. Please note that if there are any issues concerning what the dispatchers at x34567 x34567 ask you, you may ask them to “ patch” you thro ugh to their Supervisor.
Communications Center at x52222 is x52222 is your link to all Med Center On-Call people (with the exception of ISD staff)
ResNet Help Desk at x35154. Laurel Contomanolis, and other ResLife ResLife Staff may may be utilized to refer issues to the Duty Dean, Resident Advisors, or to ResNet staff when the ResNet Help Desk is not open.
If none of these dispatch points work, consult with another Manager On-Call Rep or see if the Directory's area listing ("Departments, Offices, and Services") offers contact information.
Disruptions of any voice related services in the Medical Center require communication to the Administrator On-Call via the Communications Communications Center.
Any safety issues must be immediately communicated to Security at 275-3333.
If a customer declares a situation to be an emergency, do not question that judgment. Consult with Security, x13 or 275-3333, immediately. Version 1.9 Confidential
45
Incident Management & Communication
Communications Manager The Communications Manager is responsible for managing Uni versity IT-wide and University-wide communications for University IT-wide incidents. [This is a role served during an incident and no t a job title.] He/She serves as a communications liaison to the IMOC during SEVERITY 3 incidents (defined below). The IMOC may choose may choose to serve in this role if another suitable employee is not identified. The Communications Manger must review all communications with the IMOC before releasing them, unless otherwise stated by the IMOC. In some cases, the CIO (or Assistant CIO, Other Directors) may require that all communications get reviewed by the CIO’s Office prior to deployment. Communications Manager needs to compose and deploy updates during the course of the incident. Th e Communications Manager should also send out a final message indicating the in cident is closed and o ffering a status report to affected users. [See sample text at the end of this document.] In some cases, the Communications Manager will need to provide details and in other cases, it will be necessary to remain vague. IMOC and CIO will provide guidance on this. Be sure to communicate with Becky Kingcaid/Alivin Ruiz if it is an issue that affects Wallis Hall. Becky will often re-tool general messages based on the needs of users in Wallis hall. It is a good idea to send her copy before releasing to the general public so she has a heads up. Refer to the templates and checklists provided in the Incident Management Handbook for details on communication channels, etc.
Definition Definition of Severity 3: The problem has a critical impact on key functions within the University University or its reputation. Resolution takes highest precedence.
Communications Manager Manager Responsib Responsib ilities:
Gathers details about incident.
Crafts messages messages for internal (University IT Only) and external (University-Wide) use.
Works with Office Office of Communications Communications if communication outside of the University is required. Depending on the situation, the Communications Manager may or may not be asked to speak to the press. But should never do so unless given instructions to.
Identifies appropriate communication channels.
Deploys communications according to incident timeframe through identified channels/Working with with MOC and IMOC. [All Channels]
Provides guidelines for communications communications to the Customer Service Centers and to the IT Admins so they can handle calls appropriately and deliver a consistent message.
Identifies channels for post-incident follow-up and helps prepare messages for those channels.
Retains copy of all communications communications for debrief session and for audit purposes.
If an an incident occu rs after normal business hours: The Communications Manager picks up the next business day to continue ongoing communications (internal and external) or to assist in closing out the incident. If incident is closed:
Sends final communications i
Identifies channels for post-incident follow-up and helps prepare messages for those channels.
Retains copy of all communications communications for debrief session and for audit purposes. Version 1.9 Confidential
46
Incident Management & Communication
If incident is still open:
Gathers details about incident and reviews CHRON.
Crafts messages for internal and external external use.
Identifies appropriate communication channels.
Deploys communications according to incident incident timeframe through identified channels/Working with MOC and IMOC. [All Channels]
Provides guidelines for communications communications to the Customer Service Centers and to to the IT Admins Admins so they can handle calls appropriately and deliver the same message. Identifies channels for post-incident follow-up and helps prepare messages for those channels. Retains copy of all communications communications for debrief session and for audit purposes.
Certain service disruptions require contact with general dispatch points (this is usually done by the IMOC - but you may be asked asked to continu e to provide them with updates during the course of the incident): incident):
ISD Help Desk at x53200 can x53200 can be your reference point for any ISD staff on call for desktop or Med Center department network issues, such as with Omega. If x5-3200 is closed (after hours), you will be rolled over to the Data Center (x5-9194 or x5-9195).
Energy Management at x34567, x34567, a.k.a. Customer Service Center, a.k.a. Energy Operations Center, is your link to all trades-people in Facilities. Please note that if there are any issues concerning what the dispatchers at x34567 x34567 ask you, you may ask them to “ patch” you thro ugh to their Supervisor.
Communications Center at x52222 is x52222 is your link to all Med Center On-Call people (with the exception of ISD staff)
ResNet Help Desk at x35154. Laurel Contomanolis, and other ResLife ResLife Staff may may be utilized to refer issues to the Duty Dean, Resident Advisors, or to ResNet staff when the ResNet Help Desk is not open.
If none of these dispatch points work, consult with another Manager On-Call Rep or see if the Directory's area listing ("Departments, Offices, and Services") offers contact information.
Disruptions of any voice related services in the Medical Center require communication to the Administrator On-Call via the Communications Communications Center.
Any safety issues must be immediately communicated to Security at 275-3333.
If a customer declares a situation to be an emergency, do not question that judgment. Consult with Security, x13 or 275-3333, immediately.
Version 1.9 Confidential
47
Incident Management & Communication
Sample Communication Copy General General Pointers:
Always include a heading/subject heading/subject line – even if email isn’t used. Helps Helps people get their their bearings.
Be careful not to over promise promise on a solution or quick outcome.
Provide estimates when possible.
Indicate where people can go for additional information. Use “Contact “Contact University IT” IT” in most cases – with with whatever number is appropriate for the incident.
Don’t provide too much technical technical information. information.
Speak in terms terms the average end user user will understand. understand.
Tell users users what to expect.
Keep track of all communications in a Word document and add the time the communication was sent out.
Provide updates after major attempts to solve the problem, such as server reboots, hardware swaps, etc.
Sample INITIAL Messages: Exchange Email Email Disrupt ion Between 8:30am and 10:00am today, some University faculty and staff experienced disruptions with email service. These disruptions were confined to a subset of Exchange email users. University IT support teams have isolated and resolved the issue. We apologize for the inconvenience this may have caused you and we will continue to monitor the situation throughout the day. If you have additional questions or begin to experience problems with your email, please contact University IT at 5-2000.
UNIX UNIX Email Disrup tion University IT Support teams are currently investigating issues that may be resulting in delayed email delivery. We apologize for the intermittent slowdowns you may be experiencing with email services. University IT teams are working diligently to address the issue as quickly as possible. As a precaution, our hardware hardware vendors have been called in to assist with the investigation and we will will be working with the vendors to identify actions to minimize this type of disruption in the future. We expect to provide additional information by 5:00 this afternoon (1/19). If you have additional questions, please contact University IT at 5-2000.
Sample Sampl e SUBSEQUEN SUBSEQUENT T Messages: Messages : UNIX UNIX Email Email Disrup tion – 6:15PM 6:15PM Update We are still experiencing intermittent email issues on the mail.rochester.edu mail server and we will be rebooting the server at 6:20 p.m. this evening. Mail services will be unavailable for approximately 20 minutes. We appreciate your patience as we continue to work on resolving this issue. Please continue to check back for regular updates. You can also call the University IT at 5-2000 or 3-3999 (recorded message).
Version 1.9 Confidential
48
Incident Management & Communication
UNIX UNIX Email Email Disrup tion – 7:15PM 7:15PM Update We are still experiencing email issues on the mail.rochester.edu server following the reboot performed at 6:20 p .m. As we work with our vendors to diagnose the problem, you may continue to experience intermittent availability of email. Please continue to check back for regular updates. You can also call the University IT 5-2000 or 3-3999(recorded message). UNIX UNIX Email Email Disrup tion – 9:00PM 9:00PM Update Faculty and students may still still be experiencing intermittent disruptions with with email service. Users experiencing these problems are primarily within the College. We will continue to work with vendors to isolate the source of slow email service. At this point, we are progressing through a detailed plan. We apologize for the inconvenience; we recognize the impact that this has on you and are working to remedy the remaining issues.
UNIX UNIX Email Email Disr upti on – 8:00AM Update Update University faculty, students and staff who were experiencing disruptions with email on Thursday, January 19 can now log into their email. You may experience a delay with your initial log in if you have a large quantity of unchecked messages in your mailbox. University IT staff resolved some service disruptions and is maintaining a continuous effort to address the issue. University email services will be monitored throughout the day. Please contact Information Technology Services at 5-2000 if you need assistance. Generic NetID template to be used when LDAP is disrupted. We are experiencing a service di sruption with the University’s LDAP service. This means that applications requiring a NetID for authentication are currently unavailable. IT support teams have identified what must be done to resolve the issue OR OR IT IT support teams are working to identify the cause of this disruption. [if the reason is known and can be shared in terms the users will understand, add a brief statement here] We apologize for the inconvenience and we expect to have the problem resolved by [enter info here]. here]. We will provide additional updates as they are available [or enter a specific time(s)]. time(s)] . Please contact the IT Center at 275-2000 if you have additional questions.
Sometimes, Sometimes, we think w e have fixed a problem and it comes back (or was never really really fi xed to b egin with). Here’s Here’s an example of how t o handle that. First Message – We have received new information that some University faculty, students, and staff members are still experiencing intermittent email issues. We apologize for the slowdowns you h ave been experiencing the past few days. We recognize the importance of email service and that this disruption has happened at an inopportune time. We are working diligently to restore full email services. Please contact University IT at 5-2000 if you need assistance. Status information is also posted on the IT Notices found at www.rochester.edu/its/.. www.rochester.edu/its/
Second Message – Improvements to the email environment continue. We recognize the importance of email services and Information Technology Services continues to work diligently to restore full email services. Please contact Information Technology Services at 5-2000 if you need assistance. Status information will continue to be posted on the IT Notices found at www.rochester.edu/its/. Version 1.9 Confidential
49
Incident Management & Communication
Sampl Sampl e CLOSED CLOSED Inci Inci dent Message: [ I t is important to send ou t a final communication to let users know that all has been restored and to offer an explanation of what to expect.] expect.] Exchange Email Email Disrupt ion On Friday, May 5 at 8:25 a.m., University IT became aware of an issue with one of the Exchange 2003 email servers that resulted in a brief email outage for a subset of Exchange emai l users. University IT support teams isolated and resolved the issue and had email restored by 10:00 a.m. During this time, emails were held in the queue and delivered when email services were brought back online. Please be assured that no emails were lost during this event. We apologize for the inconvenience this may have caused you and we will continue to monitor the situation throughout the day. If you have addi tional questions please contact University IT at 5-2000.
In some cases, it may be necessary necessary to provide infor mation about an incident to people NOT NOT directly affected. affected. An example communication is p rovided below. Message for IT Key Contacts (Includes IT people outside of University IT – was sent to help other IT support users who were affected by the outage) Dear IT Colleagues, On Thursday, January 19, University faculty, students and staff started to experience i ntermittent disruptions with UNIX email service on the mail.rochester.edu server. Users experiencing the problems were primarily within the College. University IT worked with our vendors to isolate the source of slow email service. Users who were experiencing disruptions with email on Thursday can now log into their email. They may experience a delay with initial log in if their mailbox contains a large quantity of unchecked messages. Processing capacity was added to help move mail through the various checkpoints (anti-spam/anti-virus). University IT staff resolved some service disruptions and is maintaining a continuous effort to address the issue and University email services will be monitored throughout the day. Regular updates will be posted online at: www.rochester.edu/its/ www.rochester.edu/its/ - IT NOTICES. Please use this information to keep your area up to date with this issue. University IT uses this area to communicate with the University community on a regular basis and will be a source of information for you on this issue, regular updates on maintenance outages, and other University IT services. If you have additional questions, please contact University IT at 5-2000.
Version 1.9 Confidential
50
Incident Management & Communication
Web Web Cont ent Hack Hack – Immediate Immediate Acti ons On receipt of WebWatcher or other notification of a hack 1. Go to the page reported and see what has happened 2. If there does not appear to be anything different check with the owner of the file. 3. If confirmed hack begin notification of …? How should we start the escalation? 4. Do not delete or mov e any any of t he hacked hacked files until the evidence evidence is reviewed 5. Is this a OS hack or a content hack? If content hack continue (We should have a procedure for assessing an OS hack) 6. Get the modified date and time of the hacked file 7. Using that time minus one hour find all files that have been modified You are looking for hack tools and any additional hacked pages. If nothing turns up use minus two hours etc. 8. Review these files for hack tools 9. Review log files for the hacked file access record and note the IP number 10. If more than one file is hacked find those in the log and capture the IP number 11. Preserve copies of the hacked files 12. Redeploy or restore the hacked file 13. Identify the ISP of the hacker and their entire IP range 14. Block that range at the router. 15. File an abuse report with the ISP of the hacker 16. File an incident report with Campus Safety 17. Evaluate the methods used & determine what actions can be taken to prevent a repeat.
Version 1.9 Confidential
51
Incident Management & Communication
Debrief Procedures Procedures 1.
Debrief will be scheduled by the scheduled IMOC, not the acting IMOC. Meaning, if a scheduled IMOC IMOC is unavailable to be IMOC and an acting IMOC is leading the incident, the scheduled IMOC will be responsible for scheduling and leading the incident debrief. If circumstances prevent the scheduled IMOC from assigned assigned duties, the acting IMOC will be responsible.
2.
Debrief should occur no later than one week after the incident, with one day after the incident preferred while information and events are fresh in participants’ minds.
3. Debrief documentation documentation is to to be stored stored in the Incident Management Management and Communication Communication SharePoint site, located: https://sharepoint.its.r https://sharepoint.its.rochester.edu/sites/ITS-IMC/Shared% ochester.edu/sites/ITS-IMC/Shared%20Documents/Forms/AllItem 20Documents/Forms/AllItems.aspx s.aspx under under the Incident Reports and Debriefs. 4. After documentation documentation is complete, complete, send an email to Bill Waterhouse. He will in turn produce produce a UR Audit update to be sent to the University’s Audit department and Julie Buehler.
Version 1.9 Confidential
52
Incident Management & Communication
Debrief Agenda Template Template Event Date: Event Time: Event Event Descriptio Descriptio n: At ten dees : Debrief Facilitator: Debrief Date:
Item 1.
Notification
Was everyone notified in a timely manner? What would have made it better?
2.
Turnout
Was everyone there who needed to be there? What other personnel would have helped?
3.
Communications
4.
Personnel
Did we communicate to each other well? Did we communicate to customers well? How can we improve the process? Did we have the correct personnel on-site throughout the incident? Was the personnel rotation correct?
5.
Equipment
Were the room(s) equipped with the correct items to support the incident? What other equipment would have helped?
6.
Intra-Departmental Cooperation
Did the University IT business units work together in the best manner?
7.
Inter-Departmental Cooperation
Did University IT work together with other University departments in the best manner?
8.
Initial Strategy
Did we use the best strategy to minimize incident timeframe? What strategies would have improved turnaround?
9.
Execution
Did we execute the strategy in the best manner? What could we have done better to improve turnaround?
10.
Clean Up
Was incident closed so everyone knew to step down from a Severity 3? Was chronology published in a timely manner?
11.
Customer Impact
What feedback did we receive from customers?
12.
Follow Up Items
What open items still need attention?
13.
Lessons Learned / Recommendations
What did we learn? What would make incident response and communication better?
14.
Audit Notification
IT Security will provide incident notification to University Audit
Version 1.9 Confidential
53
Incident Management & Communication
Updating Procedures The following procedures manual was initially developed by the Incident Management & Communications team between February and June 2006. If you have any questions, concerns, or modifications to the following following procedures, please contact the IT Center (275-2000
[email protected] [email protected])) The following people had a major role in the creation of this document: Project Sponsors
Kate Crowley
Network & Communications
Project Manager
Bill Waterhouse
Security & Policy
Project Participants
Norm Acunis Mike Fitch Karen McVige Joe Pasquarelli Jay Riley Mercedes Fredericksen Jason Wagner
Network & Communications Network & Communications Data Center – Production Control Academic Technology Applications & Architecture Office of the CIO – Communications Academic Technology – Emergency Preparedness
Others assisted with its creation, and Information Technology Services i s thankful for the participation and guidance to better serve our customers.
Version 1.9 Confidential
54
Incident Management & Communication
Change Change Control Version Number
Name Person
Section change
Description change
Date
Bill Waterhouse
IT Alert
Modified procedures to follow G2Alert alert custom list modifications
12/08 /08/200 2006
1.0
Add University IT Security to notify notify Audit of major University IT incident
12/13 /13/200 2006
1.1
Updated ISD On-Call schedule for 2007
1/3/2007
1.2
1/9/2007
1.3
7/15/07
1.4
12/10/07
1.6
02/01/08
1.7
5/29/09
1.8
6/17/09
1.9
Bill Waterhouse
Communication Checklist – Section 6 Debrief Document
Bill Waterhouse
Bill Waterhouse
B.J. Block
ISD On-Call Update
IM&C Quarterly Update – Q1 2007
IM&C Quarterly Update – Q3 2007
Appendix Bill Waterhouse IMOC Schedule ‘08
Bill Waterhouse
IM&C Quarterly Update – Q4 2007
Bill Waterhouse
IMOC Schedule IT Alert (G2Alert) ISD Manager On Call
Bill Waterhouse
Contact Information Bridge Phone #
1. ISD Conference Call # in IMOC checklist 2. Service Monitoring query in IMOC checklist 3. NCS MOC to forward Operations phone numbers if TH evacuation 4. Debrief is required, and scheduled IMOC will schedule (not acting IMOC) 5. Add Services Monitoring (Uptime) to IMOC IMOC checklist 1. Changed name from ITS to University IT 2. Updated Information Security and Policy Director to Bill Waterhouse 3. Updated contact information for Bill Waterhouse 4. Updated IMOC schedule through beginning of 2008 5. Changed debrief documentation to state that the debrief should be sent to Bill Waterhouse and he will send to audit 6. Updated email distribution lists to new naming convention 7. General editing updates Updated appendix to include Web Content Hack Procedures Updated 2008 IMOC schedule 1. Updated 2008 IMOC schedule 2. Updated 2008 ISD schedule 3. Recovered roles deleted from version 1.6 4. Updated MOC role to include University IT MOC decision point 1. Updated University IT IMOC Schedule 2. ISD notified during any Severity 3 alert 3. Updated ISD IMOC information 1. Updated all internal & external contact information 2. Added 3-3311 bridge # throughout doc
Version 1.9 Confidential
55
