|
Post by Bryce on Dec 4, 2020 10:41:23 GMT -6
As most of you know, Crystal PM cloud servers have been temporarily disabled due to an influx of ransomware attacks. On 12/03/20, after the shut down, it was discovered that ransomware had been introduced into the datacenter where Crystal servers are hosted. Crystal PM uses Netgain, a major secure data center, to handle the hosting of all of our online servers. We chose them because they are one of the biggest and most secure data centers in the United States. They have security staff 24 hours monitoring and maintaining their servers. At this time, it is not believed to have affected any Crystal PM servers. However, due to this, extra precautions are needing to be taken and multiple servers are needing to be recreated, which will extend the amount of time necessary for the system to be offline. Our initial estimate was 12pm CST on Friday, 12/04/2020. At this time, we are now being told that the system should be back online by Monday, 12/07/2020. Please be aware, this has not only affecting our cloud servers, but is occurring worldwide. Amazon's AWS servers were recently taken offline for a similar issue. Hospitals and schools server systems are currently being locked down due to attacks as well. While CrystalPM and Netgain do everything in our power to try to prevent these problems, new methods for distribution and attack are being developed every day. If you are not a cloud office, you may also be affected by what is happening. • You may not be able to import patients from web. While this service is not completely shut down, due to the amount of traffic, it may be non-functional and may cause your system to lock up and freeze. ________________________________________ • Freezing when printing invoices - this will occur if your system is configured to print the patient portal information on the invoice. To prevent this, please go to Admin - Invoices and uncheck the option to "Include information on how to access the portal. ________________________________________ • Freezing when opening CrystalPM or going to the tasks page – please go to Admin – Tasks and make sure the checkbox to “Auto Check Web Messages” is unchecked. ________________________________________ • If you use the CrystalPM template and are not able to access your records, a PDF form can be found here: sites.google.com/view/crystalpmtemplate/cpm-template?fbclid=IwAR3Uo9075P7grbPpgRcFGjdQJHN7KcU4e5fLij9DkK-qw5-8ysJ1RSQSCi8________________________________________ • You are not able to upload documents to the patient portal until this is resolved ________________________________________ • e-Dr contact lens orders cannot be sent until this is resolved ________________________________________ • The VisionWeb integration will not function properly until this is resolved ________________________________________ • Schedule Your Exam will be down until the issue is resolved. Again, we sincerely apologize for the inconvenience this is causing and are doing everything in our power to get everyone back up and running as quickly as possible.
|
|
|
Post by Bryce on Dec 7, 2020 9:32:24 GMT -6
As of our 4:30 cst (12/6/2020) status call with Netgain, they are continuing to create backend infrastructure to make sure that no further or continued attacks will occur. They are hoping for the new data infrastructure to be in place by midnight tonight. Once they are able to turn it online and test it, they will then begin the server restoration process. CrystalPM will continue to send email updates to all offices as we receive information and timeline changes. Thank you, Crystal PM If you are not a cloud office, you may also be affected by what is happening. • You may not be able to import patients from web. While this service is not completely shut down, due to the amount of traffic, it may be non-functional and may cause your system to lock up and freeze. ________________________________________ • Freezing when printing invoices - this will occur if your system is configured to print the patient portal information on the invoice. To prevent this, please go to Admin - Invoices and uncheck the option to "Include information on how to access the portal. ________________________________________ • Freezing when opening CrystalPM or going to the tasks page – please go to Admin – Tasks and make sure the checkbox to “Auto Check Web Messages” is unchecked. ________________________________________ • If you use the CrystalPM template and are not able to access your records, a PDF form can be found here: sites.google.com/view/crystalpmtemplate/cpm-template?fbclid=IwAR3Uo9075P7grbPpgRcFGjdQJHN7KcU4e5fLij9DkK-qw5-8ysJ1RSQSCi8________________________________________ • You are not able to upload documents to the patient portal until this is resolved ________________________________________ • eDr contact lens orders cannot be sent until this is resolved ________________________________________ • The VisionWeb integration will not function properly until this is resolved ________________________________________ • Schedule Your Exam will be down until the issue is resolved.
|
|
|
Post by Jennifer on Dec 7, 2020 10:58:52 GMT -6
Is that midnight tonight as in Sunday night or Monday night? Is there an ETA how long it will take Crystal to do the restoration?
|
|
|
Post by Bryce on Dec 7, 2020 11:44:34 GMT -6
That was midnight Sunday night/Monday morning. Unfortunately, there is no ETA on how long Netgain will take for the restoration. The servers are hosted by them, and they are the ones doing this restoration process.
|
|
|
Post by erica on Dec 7, 2020 13:09:46 GMT -6
Crystal PM just got off a noon CST meeting with Netgain. In order for our cloud servers to be placed back online they needed to make sure the backend infrastructure is secure enough to prevent any future attacks. Several of the network domain controllers at their location needed to be rebuilt. There is still no indication that any Crystal PM servers with patient data have been compromised. Netgain expected to have 100 offices up by this time and ran into some permission issues with the scripts created to automate the scanning and restore process. We are hoping to have offices returned online as soon as we know our servers are cleared. Our next status update is at 5pm today.
|
|
|
Post by Gonzo on Dec 7, 2020 13:51:00 GMT -6
Thanks for the update!
|
|
|
Post by Jessica T. on Dec 7, 2020 19:07:01 GMT -6
Hi! Is there any update as to when the restoration will be done?
|
|
|
Post by erica on Dec 7, 2020 20:38:45 GMT -6
Crystal PM leadership just completed our 5pm briefing with Netgain. They informed us that they have been making steady progress with getting all the cloud servers back online. In order from them to put the cloud servers online they needed to recreate their domain controllers and scrub the network. They then began deploying scans on each individual server. There are over 60 employees working at the data center around the clock getting the all the servers up and safely returning them online. There were expectations of having many of our servers being returned by noon today and they ran into some hurdles with the updated security software and deployment. Keep in mind there are 1000s of servers across several locations in their network. The ransomware attacked the domain controllers that organized the 1000s of servers. Those servers have been replaced over the last few days and they have begun checking all the individual servers for any residual traces of the ransomware. Once cleared they will be returning the servers online. Currently they are automating the checking process. They have not provided us with an exact timeline and will know more about the deployment status later tonight. At this time, they are still not able to give us an ETA, but we are hoping to start restoring servers at some point tomorrow. Common Questions WHY IS CRYSTAL PM USING NETGAIN TO HOST ITS CLOUD SERVERS? Crystal PM had a smaller company 5 years ago host its cloud and found the company was not responsive and did not offer 24 security monitoring. In 2015 Crystal PM launched an exhaustive search across all major cloud providers to find a secure data center that was more reliable and had a faster response time. Netgain was the leader in secure and scalable IT-as-a-Service (ITaaS) for healthcare. They host 1000’s of servers across Microsoft Azure, Zayo servers, and AWS (Amazon Web Servers). They also have a 99.95% uptime guarantee and a 24/7/365 dedicated care team and met all HIPAA and SSAE 18 requirements. WHY WAS CRYSTAL PM NOT MORE PREPARED? A ransomware attack affected one of the major optometry EHR companies last year. At that point we met with Netgain and they informed us they had security policies, procedures, and updates to keep that from happening to our cloud servers. WAS ANY PATIENT DATA EVER COMPROMISED? There is no evidence that any Crystal PM servers had any data compromised CAN YOU JUST MOVE ME TO THE IN OFFICE VERSION? Once your data is accessible Crystal PM can migrate you to the In Office version immediately if requested. WHAT ARE YOU DOING TO PREVENT A SITUATION LIKE THIS HAPPENING AGAIN? We are exploring all options including: changing cloud providers, backing up all essential cloud data to Crystal PM or the local office, adding 2 factor authentication, and more ….
|
|
|
Post by erica on Dec 8, 2020 10:35:04 GMT -6
We wanted to provide an update on progress today and further insight into timelines for recovery. Client restoration remains our foremost priority, and we’ve made significant progress in our recovery efforts over the last day. A key aspect to enabling client recovery is our core management tooling platform. Our team has worked round-the-clock to rebuild this platform and enable secure network connectivity to client environments. We are at a point in our recovery efforts where we are validating and fine-tuning the recovery playbook as part of finalizing the recovery efforts. As part of our recovery process, we are also running additional scans of the environment to ensure that your system is secure prior to restoring your network. Unfortunately, due to the multiple steps required to ensure the safety of your network, we expect that the recovery efforts may span multiple days, likely resulting in extended downtime or issues during that period. We recognize the impact this is having on your business and continue to work around the clock to get your environments fully operational.
We will continue to update after our status meetings with Netgain.
|
|
|
Post by Nohely on Dec 8, 2020 11:25:46 GMT -6
Hi, thank you for the updates!
Any way our office can access any type of information, whether it be just the weekly schedule? That would be incredibly helpful.
|
|
|
Post by Gonzo on Dec 8, 2020 11:27:34 GMT -6
If on the cloud there isn't - We use the cloud going forward we are going to print a weeks schedule in advance to a PDF file that we update at the end of each day so we have a weeks worth of a schedule to view... perhaps Crystal will create a report feature to support that.
|
|
|
Post by erica on Dec 8, 2020 12:34:02 GMT -6
Crystal PM had a meeting this morning with the CEO of Netgain. He said he still hopes to have some servers returned online today. We have a meeting with the tech working directly with our cloud servers later today and we will hopefully get more insight.
|
|
|
Post by Vision NYC on Dec 8, 2020 14:07:01 GMT -6
Its coming up on 3 pm EST. Any reason to believe it'll be up and running for work tomorrow?? We are just trying to plan our shifts for tomorrow.
|
|
|
Post by erica on Dec 8, 2020 19:59:09 GMT -6
Cloud Status Update – Tuesday 7pm – Our direct contact at Netgain has informed us to be prepared for continued downtime tomorrow. They have made progress on the network infrastructure and plan to spin a few servers up tonight and in the morning. We are cautiously optimistic about this because we have been told this before. They are running scanning tools on all servers to make sure they are safe. Tomorrow Crystal PM Technical support staff will begin contacting offices to get new security credentials for your logins. Additional Common Questions: Are Just Crystal PM offices affected? No, all cloud servers at Netgain are currently offline. Netgain hosts cloud servers for Hospitals, several other EHR companies, health systems, post-acute providers, medical device and drug manufacturers, insurers, and medical groups.
|
|
|
Post by Britt on Dec 9, 2020 8:45:09 GMT -6
Is there still no ETA? Also, will we be getting new cloud connections?
|
|