Post by jessicalynsy on Dec 15, 2020 14:35:39 GMT -6
For those that do not know me, I am Mark Crowley, the Sales Director and 1/3 owner of Crystal PM. Let me take this opportunity to sincerely apologize for this situation. Not in our wildest nightmare could we have imagined a situation like this would occur. We know this has had dramatic effects on offices and are trying to do anything possible to help your practices recover. I know affected offices have been following our status updates and I will do my best to keep this message as concise as possible.
Background
In 2015 Crystal PM did an extensive search for a new cloud provider. Prior to 2015 Crystal PM had a smaller company that was not as responsive and did not offer 24/7 monitoring. We launched an exhaustive search with several Cyber Security Professional and Data Centers. We learned we needed to find a secure data center that was HIPAA compliant had SSAE 18-certified and experience in Optometry. One of our limitations with cloud was integrations with optometric machines and we wanted to find a company that had experience with integrating equipment to the cloud. Through our search we decided that iManaged was the best provider that met all our criteria. They had experience hosting with several of our competitors and were the only data center that attended Vision Expo East/West.
In 2019 iManaged was acquired by Netgain. They are a much larger entity and promised additional resources and staff. Their data center focuses on health care and had several Hospitals and Medical Companies as customers.
In 2019 one of our major competitors had ransomware on their cloud where offices were down and some offices lost data. At that point we contacted Netgain and they informed us that worst possible scenario if an office was compromised would be downtime for 24 hours and a restored backup within 4 hours of the incident.
Incident
At the end of November, Crystal PM had trouble tickets in with Netgain about the servers running slower than normal. They informed us there was a brute force attack and they looking into solutions to speed up the servers. On Thursday, December 3rd without notice Netgain stopped all incoming connections to our servers. At that point all offices and Crystal PM were unable to access any of our systems.
We were informed there was an attack on Netgain and they hoped to have servers up within 24 hrs. We were then informed there was a ransomware attack on the network that affected their domain controllers (computers that control the users and passwords of all the computers).
They found out over the 1st weekend they would have to rebuild their network in order for us to get connected again. We were told it would be the next day – then the next day – through the week. Netgain informed us on Saturday of last week that they would have a batch of roughly 65% of our servers up by 5pm on Sunday with the rest restoring soon after.
The 5pm deadline on Sunday got delayed to 11:30pm Sunday evening. At that point CrystalPM.com and ScheduleYourExam.com – both hosted through Netgain went online. They informed us that customer support would be contacting us shortly with access to the store servers. We had staff in the office waiting all night for this call, so we could have the offices ready by the morning. At 4am we called Netgain upper management and they informed us they were working on an internal routing issue, and the stores would be up ASAP. Since that point Crystal PM has had staff 24 hours at our office in preparation to set permissions and give access to offices.
Where We Are Now
After 40 hours we are still in limbo waiting for the credentials. Crystal PM can confirm that servers are active for many of our offices. 3rd party applications began syncing and sending data Sunday evening the same time crystalpm.com and scheduleyourexam.com were restored online. Crystal PM has consulted with several lawyers about taking action and an immediate injunction to get access to the data. Lawyers informed us that when we proceed with that route all communication will be exchanged through lawyers with Netgain and Netgain staff will not continue working on our servers.
Concerns
For the doctors concerned with data breach, the attack was against Netgain and their infrastructure. The FBI has been notified. Netgain informed us there is an official forensic file that they will share with us once Netgain has all severs restored that show zero files were encrypted or exported from Crystal PM servers. We will publish the forensic file to offices when available.
Compensation
Crystal PM will do everything in its power to make sure everyone is fairly compensated. Please address all claims and questions regarding compensation to Ken Richey at ken@crystalpm.com.
Moving Forward
Many offices have a local version of Crystal PM installed to help them run their practice until the cloud is restored.
We have met with over 20 cyber security specialists and data centers over the past week. In that time we learned that no data center or cloud based software can honestly say they won't be attacked. In the past 48 hours the Department of Treasury and Google have both experienced outages and potential data breaches. What analysts can tell you is that recovery should never take this long. Crystal PM is exploring all options right now. We will never let one of our offices and ourselves be this helpless again.
One of the features we plan to immediately implement is a local backup overnight of all essential data. What this means - for those on the cloud - they can either have all their essential data (patients, invoices, records, orders, appointments) sent to the office or a secure location (dropbox, house, ...) every night. If there is ever a future time when a cloud goes down or the internet quits working, an office will always be able to load a local version and be able to see and treat patients. We are hoping to have this deployed following Christmas break.
Background
In 2015 Crystal PM did an extensive search for a new cloud provider. Prior to 2015 Crystal PM had a smaller company that was not as responsive and did not offer 24/7 monitoring. We launched an exhaustive search with several Cyber Security Professional and Data Centers. We learned we needed to find a secure data center that was HIPAA compliant had SSAE 18-certified and experience in Optometry. One of our limitations with cloud was integrations with optometric machines and we wanted to find a company that had experience with integrating equipment to the cloud. Through our search we decided that iManaged was the best provider that met all our criteria. They had experience hosting with several of our competitors and were the only data center that attended Vision Expo East/West.
In 2019 iManaged was acquired by Netgain. They are a much larger entity and promised additional resources and staff. Their data center focuses on health care and had several Hospitals and Medical Companies as customers.
In 2019 one of our major competitors had ransomware on their cloud where offices were down and some offices lost data. At that point we contacted Netgain and they informed us that worst possible scenario if an office was compromised would be downtime for 24 hours and a restored backup within 4 hours of the incident.
Incident
At the end of November, Crystal PM had trouble tickets in with Netgain about the servers running slower than normal. They informed us there was a brute force attack and they looking into solutions to speed up the servers. On Thursday, December 3rd without notice Netgain stopped all incoming connections to our servers. At that point all offices and Crystal PM were unable to access any of our systems.
We were informed there was an attack on Netgain and they hoped to have servers up within 24 hrs. We were then informed there was a ransomware attack on the network that affected their domain controllers (computers that control the users and passwords of all the computers).
They found out over the 1st weekend they would have to rebuild their network in order for us to get connected again. We were told it would be the next day – then the next day – through the week. Netgain informed us on Saturday of last week that they would have a batch of roughly 65% of our servers up by 5pm on Sunday with the rest restoring soon after.
The 5pm deadline on Sunday got delayed to 11:30pm Sunday evening. At that point CrystalPM.com and ScheduleYourExam.com – both hosted through Netgain went online. They informed us that customer support would be contacting us shortly with access to the store servers. We had staff in the office waiting all night for this call, so we could have the offices ready by the morning. At 4am we called Netgain upper management and they informed us they were working on an internal routing issue, and the stores would be up ASAP. Since that point Crystal PM has had staff 24 hours at our office in preparation to set permissions and give access to offices.
Where We Are Now
After 40 hours we are still in limbo waiting for the credentials. Crystal PM can confirm that servers are active for many of our offices. 3rd party applications began syncing and sending data Sunday evening the same time crystalpm.com and scheduleyourexam.com were restored online. Crystal PM has consulted with several lawyers about taking action and an immediate injunction to get access to the data. Lawyers informed us that when we proceed with that route all communication will be exchanged through lawyers with Netgain and Netgain staff will not continue working on our servers.
Concerns
For the doctors concerned with data breach, the attack was against Netgain and their infrastructure. The FBI has been notified. Netgain informed us there is an official forensic file that they will share with us once Netgain has all severs restored that show zero files were encrypted or exported from Crystal PM servers. We will publish the forensic file to offices when available.
Compensation
Crystal PM will do everything in its power to make sure everyone is fairly compensated. Please address all claims and questions regarding compensation to Ken Richey at ken@crystalpm.com.
Moving Forward
Many offices have a local version of Crystal PM installed to help them run their practice until the cloud is restored.
We have met with over 20 cyber security specialists and data centers over the past week. In that time we learned that no data center or cloud based software can honestly say they won't be attacked. In the past 48 hours the Department of Treasury and Google have both experienced outages and potential data breaches. What analysts can tell you is that recovery should never take this long. Crystal PM is exploring all options right now. We will never let one of our offices and ourselves be this helpless again.
One of the features we plan to immediately implement is a local backup overnight of all essential data. What this means - for those on the cloud - they can either have all their essential data (patients, invoices, records, orders, appointments) sent to the office or a secure location (dropbox, house, ...) every night. If there is ever a future time when a cloud goes down or the internet quits working, an office will always be able to load a local version and be able to see and treat patients. We are hoping to have this deployed following Christmas break.