You are responsible for ensuring that all systems and equipment are at the minimum revision levels that we have certified for use with Verizon’s VoIP service.
Monitoring and Management
Verizon uses world-class Unified Communications application management platforms to provide both reactive and proactive application monitoring, trouble alerting, trouble identification, trouble resolution and remediation.
The following is monitored with a default polling rate of 30 sec:
- VM health (CPU, Memory, Disk, etc)
- UC Application health (critical service crash, DB health, device registration drop threshold, media resource exhaustion, route (SIP trunk) exhaustion, license exhaustion, etc)
If a critical event is discovered (ex: High CPU, critical service crash, memory exhaustion, VM down, etc…) an alert is automatically generated. Alarms go to the service monitoring tool which automatically creates proactive service tickets. The proactive service tickets get assigned to support engineers on duty for a resolution.
Proactive notifications are only sent for issues within the main applications (call manager, IM/P and voicemail).
Patch Management
Verizon maintains overall responsibility for patching. The customer will receive security patches and bug fixes via regular periodically scheduled release management, within versions that are available and certified for use in HCS deployments. Patches will stay within current versions of certified HCS releases as a general practice to ensure that only the most secure software available for the solution is being utilized. Emergency patching will be applied as necessary. You may request a specific patch application, if required, due to a specific functionality issue in your environment, and it will be considered, but not guaranteed, for use.
There are defined operational procedures in place, and Cisco’s publically stated vulnerability policies are here:
https://tools.cisco.com/security/center/resources/security_vulnerability_policy.html#asr
Certificate Management
UCCaaS certificates were installed during the installation process process with your cooperation. To ensure your certificates continue to function properly we will periodically need your organization’s assistance to renew them. Certificates typically expire in one, two or three year intervals.
Certificates are used to validate the identity of your organization and are used to insure secure communication between devices at your location(s). Since certificates require use of your domain, Verizon is unable to purchase these certificates on your behalf.
We will provide multiple CSRs (Certificate Signing Requests) to you whereupon you should have the certificates signed using your normal processes. Certificate signings are usually handled by the Security or IT teams within your organization. We need you to provide the email address of a contact within your organization that will be responsible for this process and we will provide these CSRs directly to this contact. You will need to pass the signed certificates back to us so we can install them within your UCCaaS environment.
Please note, some certificates, for example certificates used with single sign on, are not exclusively used with UCCaaS or managed by Verizon. Therefore additional customer internal steps may be required to manage these certificates.
Preguntas frecuentes
- Why do I need a certificate? A certificate validates the authenticity of the servers used within the UCCaaS service architecture, insuring secure communications between devices and endpoints. Third party certificates are required for activities using encryption such as Expressway since this is further proof of the integrity of the product.
- What happens if I don’t have the certificates signed? If the certificate is allowed to expire, the end-user will receive a warning that their connection is not secure or is not trusted each time they log in to Jabber. While the warning message can be bypassed, this end-user experience is likely to be considered annoying and distracting by most organizations.
- What do I need to do? Please provide the email address of a contact within your organization that will be responsible for the signing of certificates and we will provide the necessary CSRs directly to that contact. The certificates should be signed using your usual processes and then passed back to us after signing. The signing is usually handled by an IT or security technical team within your organization.
- What will be the end-user experience when the certificates are updated in the data center?
a. End-user devices will reboot.
b. If PCs are daisy chained to the phones they will experience a very brief interruption in service but, PCs should not reboot.
c. Please note, the certificate is applied at the cluster level, so we can’t schedule how/when individual phones will reboot. However, the application of the new certificate will be coordinated with your organization to insure it is done at a time that works best for you.
d. Users who are using Jabber will have to log back in after the certificate is applied, and they will be prompted to accept the new certificate.
Please note, phones which do not have the security necessary to support a certificate would be excluded from this process. These phones will often be end of support as well.
Ongoing Capacity Management
Verizon reviews capacity across a variety of software and hardware elements to ensure optimum performance for all data centers in all regions. Factors such as computing power (available CPUs), available storage, licenses, bandwidth, call capacity etc are analyzed to determine which factors need short and long term attention. Appropriate remediation is then scheduled and executed based upon the analysis.
Upgrades
Verizon will perform up to one major software upgrade of the UCCaaS application and features per year, if available and as specified in the upgrade SOR. Such software upgrades will be communicated to, and wherever possible, coordinated with Customer. UCCaaS applications and features versions must be kept to within 2 versions of the most current version supported by the UCCaaS datacenter (example: If UCCaaS platform is at 11.x, Customer version cannot be less than 9.x). It is the customer’s responsibility to ensure that Customer maintained applications, devices, hardware and CPE are compatible with the most current version supported by the UCCaaS datacenter. If Customer does not maintain the software upgrade of the UCCaaS application and features, Verizon may terminate Customer’s Services for cause, upon written notice to Customer.
Please note that Verizon prefers to upgrade at the X.5 version of software, to allow time for early bugs and concerns to be removed. Additionally, we test releases on internal versions of the platform prior to customer release.
Please see the “UCCaaS Upgrade Experience” deck for a complete overview of the upgrade process and your responsibilities. A simple overview of the steps required during an upgrade is as follows:
Cisco UCM implements the concept of active and inactive partitions. When performing an upgrade of CUCM, the upgrade is performed on the inactive partition. CUCM does this by copying the configuration database and applying the new version of the software on the inactive partition. This allows the “upgrade” of the platform to be done anytime, even during business hours. Once the upgrade has completed in the background, all that remains is a reboot of the platform to then begin using the new version of the software. In the interim, there will be the need to impose a change moratorium to ensure that the configuration database doesn’t change from the time the upgrade begins until the time the swap to the newly active partition is performed.
During the reboot of the publisher the customer will experience no service outage, except the inability to make platform changes until the upgrade has fully completed. When the subscriber servers are rebooted, the customer will experience brief, intermittent periods where phones will failover between the active and backup subscribers. For this reason, it is recommended that the swap to the active partition be done outside of regular business hours for the customer.
Ongoing Maintenance Process
We will periodically have to perform maintenance on your service to ensure the best possible experience.
A maintenance event occurs when a change to the UCCaaS platform is necessary to update critical system infrastructure and it supports our ability to continuously provide high-availability service to the entire base. As part of preparation for maintenance events, all work is approved by a collective change board comprised of Network, Data Center, Operations and Product teams.
Verizon has official blackout dates over Thanksgiving weekend as well as an extended black out period over Christmas and New Year’s. During this time we do not perform network changes or maintenance, and any emergency event[CJE1] requires an executive level approver before it can be performed.
Unofficially, we do not perform maintenance over major public holidays, specifically over Memorial Day, Fourth of July and Labor Day. Maintenance over these dates do not require executive approval, but we avoid these dates except for emergencies so as not to disrupt customer networks.
Verizon is not able to accommodate individual customer schedules for maintenance events which impacts multiple customers. Verizon can usually be flexible on the timing for any customer specific only maintenance.
Please note, customers who have just been on boarded may not be notified of scheduled maintenance since the announcements may have preceded the onboarding. While we will work to notify all customers, we cannot guarantee the creation announcements for the one month following onboarding.
Note that HCS version upgrades do not follow the below detail, but are rather scheduled individually by a DECCx Delivery Coordinator as part of your project.
The below outlines our process and definitions.
Maintenance activities are assigned a L1, L2 or, L3 classification. The classification impacts your experience.
L1 – Customer Affecting/High Risk Activities
- Maintenance occurs between 12am-6am local time of the data center being impacted
- L1 maintenance includes anything which causes an interruption of the following services: calling (Call Manager), voicemail (Unity) and instant messaging and presence (Unity Connection) and Attendant Console.
- Please note, any event which requires a re-registration of the above services due to an outage/failover is automatically considered an L1.
- At times, maintenance events may require exceptions to the above standard regarding advance notice:
- Emergency maintenance events may require less than ten days advance notice to customers.
- At times, maintenance events may require exceptions to this standard with regards to the duration of the outage:
- Maintenance events in which the time to complete the work exceeds the standard hours of 12am–6am.
- In case of lengthier maintenance events, we will attempt to complete the event at 11pm – 7am local time of the data center being impacted, but endeavor to never start before 10pm local time of the data center being impacted.
L2 – Minimal Customer Affecting Activities
- Maintenance occurs between 10pm-7am local time of the data center being impacted
- L2 maintenance includes any outage to non-mission critical services such as: Corporate Directory, Federation, Extension Mobility, Administrative Portal, IPMA etc…
- L2 maintenance events require notification 48 hours in advance to customers
- Please note, phones may re-register due to maintenance of the above services. Since calling, voicemail, instant messaging and presence and attendant console are not failing over, the maintenance is still considered an L2
L3 – Low Risk activities
- Maintenance occurs 5PM - 6AM local time of the data center being impacted
- L3 maintenance has no interruption of customer’s service or traffic
- Scheduled at least 48 hours in advance but Customers are not notified.
Communication for Maintenance Events
Verizon’s standard format will include the following outline of events. Please reference the Welcome Kit section on “Business Continuity Planning”, which includes expected behavior during failover events.
Maintenance notifications will be sent prior to maintenance events via the MASTARS system and the ETMS ticketing system.
The format of the ETMS notifications will be approximately as described below. When time permits Engineering will also send out ETMS notifications at the start and stop of events, as well as if event has been cancelled.
Verizon Maintenance Notification
Verizon UCCaaS Customer:
This email is to inform you of a maintenance event being done on our UCCaaS network. Below you will find detailed information on the maintenance and any impacts to your environment. If you have additional questions regarding this maintenance, please contact your Verizon Account Team or Service Manager.
Thank you for your business and your partnership.
Verizon Support Team
Maintenance Date/Time (GMT): | Dec 28 2021 05:11 GMT - Dec 28 2021 06:11 GMT |
Maintenance Location: | BELTSVILLE, ID, US |
Impact Duration: | 1 Hour(s) |
Verizon MASTARS Request number: | XXXXXXXXX |
Verizon ETMS Ticket number: | 2018xxxxxxxxxxxxxxxxxx |
Maintenance Category:
Maintenance Category | Impact |
---|
Infrastructure & Network Maintenance | Yes / No |
UC&C Applications | Yes / No |
Management Applications | Yes / No |
Maintenance Impact
Service Impact Area | Without Expressway Impact | With Expressway Impact |
---|
Audio & Video Calling services | Yes / No | Yes / No |
IM/P services (Jabber) | Yes / No | Yes / No |
Buzón de voz | Yes / No | Yes / No |
Administrative Portal | Yes / No | Yes / No |
Attendant Console Standard and Advanced | Yes / No | Yes / No |
Corporate Directory | Yes / No | Yes / No |
Extension Mobility (hot desking / hoteling) | Yes / No | Yes / No |
IPMA (Manager Assistant with Shared Line Support) | Yes / No | Yes / No |
Integrated Virtual Machine Management | Yes / No | Yes / No |
Off-Net Video Calling | Yes / No | Yes / No |
Inter-Domain Federation (instant messaging outside of organization) | Yes / No | Yes / No |
Third Party Applications | Yes / No | Yes / No |
For the most up to data Business Continuity information, including impacts to specific services during maintenance and outage events, please see the UCCaaS Welcome Kit:
http://www.verizonenterprise.com/Support/welcome-kits/uccaas/