|
Post by ecastaff8 on Dec 9, 2020 11:26:15 GMT -6
At our office, we trust Crystal PM staff is doing all they can to keep us informed as they receive information from Netgain. Thank you for those who are working around the clock to resolve this matter!
|
|
|
Post by OpsecMan on Dec 9, 2020 12:35:23 GMT -6
Pro tip for Crystal PM; have 2 data centers hosted by 2 separate companies. Sincerely, OpsecMan
|
|
|
Post by Jenny12920 on Dec 9, 2020 12:42:10 GMT -6
As of noon today we do not have any cloud servers online yet. We are still awaiting an update from Netgain on the progress made. Who is helping Netgain get the servers up: Netgain has been working with third-party cybersecurity and forensic experts from Charles River Associates. www.crai.com/service/cybersecurity-and-incident-responseSeveral offices has expressed interest in moving to the In Office version of Crystal PM when the data is available. For those offices we can install a blank server version of Crystal PM in their office today and once the cloud servers return online get the data transferred. All offices wanting to move the In Office version will receive 2 months free and then: 1) Switch back to cloud version 2) Continue with In Office Lease of Crystal PM at substantially reduced rate 3) Purchase In Office Crystal PM at substantially reduced rate View AttachmentHow do we go about signing up to get the In Office discounts and such?
|
|
|
Post by Lady on Dec 9, 2020 12:51:51 GMT -6
^She asked how to sign up for the In Office discounts....Read everything before posting something ignorant.
|
|
I have an office to run
Guest
|
Post by I have an office to run on Dec 9, 2020 13:09:38 GMT -6
So, should I start looking at how I can get compensated for this inconvenience.. or are you giving something else besides 2 months of free blank canvases?
|
|
|
Post by StuckinSouthernOK on Dec 9, 2020 13:42:24 GMT -6
My question for the moderator: Why has it taken you this long to offer to install the in office version of Crystal for all of your clients who are currently stuck without access to the cloud version? This has been down for almost 5 business days, and while I understand that the in office version is not a fix all to this current issue, it would have been nice to have this offered to us at the beginning of this week and not 3 days into the week? Also, for those of us interested in switching to the in office version of Crystal, how long will the transfer of all patient/practice data take to complete? Will we be looking at another extended downtime, or will this take place after close of business/ on the weekend? In addition, could this be a backup service for any existing clients, so that in the event of something like this happening again, we are not stuck in the same boat?
|
|
2020 was to be our year.
Guest
|
Post by 2020 was to be our year. on Dec 9, 2020 14:13:57 GMT -6
I hate to say that this is worse than the covid shut down, but it feels like it is. This is also a public relations nightmare for Netgain and Crystal. I encourage Crystal to give us some short and long term solutions even hourly to help us know something is being done. Hints from other offices as to how they are making do would be another help. We are using Solution Reach to try and recreate the schedule, calling the lab for previous RXs and lens types and previous contact lens RXs, and getting exam reports from specialists that we referred to help get past histories. Sharing other hints may help each of us to at least function until Crystal is up again.
|
|
|
Post by LORI96 on Dec 9, 2020 14:22:31 GMT -6
On the above text that says you can install a blank server version of CPM... Can you explain what that looks like? I am guessing none of our prior data will be in it- We won't be able to process credit cards like we had been doing before attack?
Is there any back up data of our information that we can have sent to us? What guarantee do we have that our records and information from CPM is still available?
Are we to guess by your mentioning a 2 month free In office version that you think it will be 2 months before we can access our information?
We are sitting ducks at our office. The end of the year.. This is the WORST time to have NO access to our examination schedule or patient records. We can't access prescriptions to sell glasses or contacts or anything that pertains to patient information.
|
|
|
Post by OpticianMA on Dec 9, 2020 14:41:26 GMT -6
Hi Lori96 - Not an official Crystal employee but I would image its going be exactly like Crystal you know and love but with zero data... this includes your billing codes (hope you made made a backup), office details / locations, Lab services, employee logins etc. I don't think this will take 2 months but they are offering that so there isn't a mad rush once Crystal is back to have the data from the stand alone imported. (?)
Again I'm just guessing here, perhaps someone from Crystal can clarify and provide the best way to start this in office stand alone setup.
|
|
|
Post by erica on Dec 9, 2020 15:14:16 GMT -6
As of noon today we do not have any cloud servers online yet. We are still awaiting an update from Netgain on the progress made. Who is helping Netgain get the servers up: Netgain has been working with third-party cybersecurity and forensic experts from Charles River Associates. www.crai.com/service/cybersecurity-and-incident-responseSeveral offices has expressed interest in moving to the In Office version of Crystal PM when the data is available. For those offices we can install a blank server version of Crystal PM in their office today and once the cloud servers return online get the data transferred. All offices wanting to move the In Office version will receive 2 months free and then: 1) Switch back to cloud version 2) Continue with In Office Lease of Crystal PM at substantially reduced rate 3) Purchase In Office Crystal PM at substantially reduced rate System Requirements.pdf (132.15 KB) Our support staff is calling every cloud office and will be able to answer your questions. Thank you for your patience as we work though everything.
|
|
|
Post by erica on Dec 9, 2020 19:50:04 GMT -6
Update Wednesday 6:30 PM Crystal PM was just informed by Netgain that they are still working on core infrastructure services and systems are being focused on at this point, they need to get their core systems up and running to ensure successful restoration. They have hit some setbacks with the recovery process. At this point we are asking offices to prepare for the cloud servers not to be restored until this weekend. We will send an official email out tonight with more information when available. If you have not signed up for the email status updates, please click here eepurl.com/gksMjb
|
|
|
Post by Bryce on Dec 10, 2020 11:47:57 GMT -6
Cloud Status Update – Thursday 12:00pm CST
First, let me say our sincerest apologies for the difficulties this situation has put on offices. Not in our wildest nightmares would we think that we would have an outage to our cloud that could last a week. It is unacceptable and know Crystal PM is trying everything possible to get your offices back up and running. Netgain has stopped communicating directly with its customers and are only sending daily status emails out at this point. Our last message received is at the bottom of this email. We have been told our offices are on a priority list with Netgain.
Crystal PM has requested access to our cloud server backups. Netgain backups up all servers every 4 hours with a virtual image that can be restored. Netgain informed us that all backups on in their network and all resources are being used to get the entire network restored.
Crystal PM has met with numerous cloud consultants and cyber security experts over past week. The standard line is that no cloud company can be guaranteed to remain up (even amazon and google have been down) – but there are numerous additional security measures that can be taken. Keep in mind there is no evidence in this situation that any of Crystal PM’s security measures caused this outage.
Crystal PM will never put an office into a situation like this again. At this moment, our developers are working on an essential backup that can securely transfer the cloud data locally either to the office or to one of Crystal PM’s secure servers every night. This will include all patient data, scheduling, financials, medical records, and frame/CLS orders. This will allow an office with an internet outage or major cloud collapse access to all essential data. If it is a minor outage, you can use it as a read only account of the office data. If it is a major outage, you can use the local backup version and then sync the data back to the cloud after the outage is over.
Update from Netgain:
We are continuing to work through the process of containment and service restoration in concert with our cybersecurity and post-incident recovery experts. Our singular priority is to get services restored in a secure manner and limit any further threat. We are leaving no stone unturned in that process.
Where needed, we have validated that we have clean data backups for all impacted clients.
The work ahead of us is broken into four primary phases, three of which are happening in parallel:
• Restore shared services: We have several shared services used by our clients (including remote desktop environments and email gateways) that were impacted by this incident. A dedicated team is working around the clock to get this key environment fully operational as it will enable functionality for many services across the network.
• Restore management console access to client environments: We have a centralized management platform that provides us with both automation and visibility across client environments. Since we had to take extraordinary steps to limit the spread of the attack last week, we are carefully restoring access across all client environments.
• Conduct complete environment scans: We have worked with our third-party cybersecurity experts to develop a comprehensive (and automated) checklist that we will execute across each client environment as part of validating that we have removed all traces of the threat. Where required, we may have to rebuild specific servers and restore data from backups. In conjunction with the steps above, we are actively working through this process.
• Hand-off client environment for validation: We will be working closely with you to hand-off the environment, support any validation activities, and fine-tune the end-point protection solutions enabled on your network.
I realize that the big question you have is the exact timeline for service restoration. This will vary by client given the various factors at play, preventing us from pegging a specific timeframe. Our teams are working without pause given the disruptive nature of this impact, both to your business and ours.
|
|
|
Post by erica on Dec 10, 2020 21:20:35 GMT -6
Thursday 9pm Crystal Practice Management just spoke with the CEO of Netgain this evening. He informed us that the network infrastructure of their cloud has been rebuilt and they are now beginning the process of getting individual cloud servers back online. As the servers come online they will be installing new security software which will cause several reboots. When the servers are tested and ready for use you will be contacted by Crystal PM to give you the go ahead to begin using them. Tomorrow, we hope to have a better ETA on when these servers will be available. Netgain also committed to do a better job with communication with more frequent updates.
|
|
|
Post by jessicalynsy on Dec 12, 2020 8:31:49 GMT -6
Crystal PM is still awaiting notifications from Netgain on the status of our servers. Netgain claimed that they are spinning up servers as we speak, but we are unaware of any of the servers being put online yet. We are losing confidence in their timeline. We are expecting a status update this evening and also have a status meeting set for noon today with them. Crystal PM will be sending out information as it becomes available.
|
|
|
Post by Bryce on Dec 14, 2020 12:20:18 GMT -6
Cloud Status Update - Monday Morning
The 1st batch of clouds servers were restored online last night at midnight including online forms/portals and schedule your exam servers. CrystalPM has still not received updated credentials to login to those servers. We have been notified 3rd party companies have been syncing data. Our staff was standby all night in anticipation of being able to setup new credentials and notify offices. As soon as CrystalPM receives connection to the servers we will update credentials and notify offices.
|
|