A number of serious security vulnerabilities affecting many CPU architectures were disclosed this week (CVE-2017-5753, CVE-2017-5715, and CVE-2017-5754) by Google’s Project Zero team and others. Our team is working with vendors and our own engineers to determine the implications for our platform, but the expectation is that a fleet-wide reboot will be necessary to protect against these issues.
While we work through our response plan, please understand that due to the nature and seriousness of these issues a rapid response may be necessary. As always, we will provide as much advance notice as possible. We will communicate directly to you, with scheduling information, if a reboot of your Linodes is necessary.
We will continue to keep you updated here as more information becomes available.
Information regarding these vulnerabilities can be found on the following sites:
- Intel’s response
- Google’s Project Zero post
- meltdownattack.com
- Documentation/x86/kaiser.txt
- LWN: KAISER: hiding the kernel from user space
Update: January 4, 2018
We are continuing to investigate this issue and wanted to provide a brief update as to where we are:
- We are postponing all unrelated maintenances to focus our efforts and resources on mitigating this issue.
- As discussed by the Scaleway team earlier today, due to the incomplete information provided by hardware manufacturers, we joined forces with other potentially impacted cloud hosting providers including Scaleway, Packet, and OVH. We’ve created a dedicated communications channel to share information and work together to address the Meltdown & Spectre vulnerabilities.
- We are continuing internal evaluation and testing of mitigations.
- We have discussions set for tomorrow for a deeper dive with the hardware providers.
We will continue providing updates here as appropriate.
Update: January 5, 2018
We’re continuing to make progress, and wanted to share the latest with you:
- The latest stable and longterm Linux kernels were released today with the KPTI / Meltdown patches in place. As such, we have made the 4.14.12 kernel available to you and have set it as the latest. If you are leveraging a Linode kernel, upon your next reboot your Linode will be upgraded to this version. This doesn’t fully mitigate you from the Meltdown and Spectre vulnerabilities, but provides us a good foundation to work with while planning for full remediation.
- We’ve had planning sessions with our hardware providers and have been working on implementation plans for kernel, hypervisor, and firmware updates. All of these will be required to get us into a remediated state, but not all of these are available.
We are not expecting much movement on this over the weekend while we wait on external dependencies, but will certainly provide updates here if there is. If not, more updates will be shared here on Monday next week.
Update: January 8, 2018
We are continuing to make progress with our internal testing, but are still waiting for microcode updates from our hardware providers. Both the microcode update and the kernel update are required in order to ensure there is proper mitigation to the three variants of Meltdown and Spectre.
Update: January 9, 2018
We spent today preparing the plan for deploying the Meltdown mitigations across Linode’s fleet. Over the course of the next day we will be implementing fixes to a subset of the fleet, monitoring for impact, and then continuing the rollout to the rest. The Meltdown mitigation requires reboots of our physical hardware which will reboot the Linodes hosted on them. A subset of Linodes in the Tokyo 1, Frankfurt, and Singapore data centers will be rebooted as part of this initial group. For those affected, you will receive a support ticket and email with scheduling information.
The reboots for this week address Meltdown only. We have testing and planning occurring in parallel to address Spectre. Additional reboots over the coming weeks will be required to properly mitigate all Spectre variants.
Update: January 10, 2018
The rollout to the subset of our fleet has gone well so far for the Meltdown mitigation. We are continuing with this plan and will be conducting reboots over the course of the next several days for the rest of the fleet. Customers affected will receive support ticket and emails with the reboot window for their Linodes with 24 hours of minimum notice.
- Due to the ongoing nature of this issue, the following status page has been created.
- We are posting a document soon that better talks about Meltdown and Spectre to show what it means to you and what you can do to prepare for it on your Linodes. We’ll share a link to this on an upcoming blog post.
Update: January 11, 2018
The mitigation process is continuing forward for Meltdown and we’re making progress each day across the fleet. There is a new guide available with more information on these vulnerabilities and how you can protect your Linode: What You Need to do to Mitigate Meltdown and Spectre.
Update: January 12, 2018
We are continuing forward with the Meltdown mitigation process and have reboots scheduled over the weekend. Our schedule runs through January 18th. We are going to pause our daily blog updates until this is complete unless other actionable news becomes available.
Update: February 8, 2018
As a reminder, all of our KVM hosts are now properly mitigated for Meltdown. We are continuing to work towards a proper mitigation for the Spectre vulnerability and will be providing an updated plan on our blog once one is available.
For more information on these vulnerabilities, the status of our fleet, and how to protect your Linode; please refer to the Meltdown & Spectre guide.
Comments (77)
Wishlist: alternate-CPU-architecture Linode hosts 😉
-Eugene
Sadly, portions of this affect both AMD and Intel, and likely others, fwiw.
My wishlist would be that all VPS providers could get the same early notifications and patches that a certain trendy yet slower provider received two weeks ago.
Yup, ARM’s response is https://developer.arm.com/support/security-update
fwiw, Scaleway is patching & mass rebooting hypervisors 1/4/2018
Sadly, it appears that even ARM CPUs are affected (to some degree) by this design flaw – so even if other CPU architectures were available, a reboot would still likely be necessary.
I could be wrong, but I think Eugene is referring to an alternative architecture like RISC-V…?
https://www.codasip.com/2016/09/22/what-is-risc-vwhy-do-we-care-and-why-you-should-too/
Thanks for the update. One more question, the patch says 5-30% performance hit depending on workload. Do we need to add more VMs to deal with the load?
@Scott, @Eugene – it seems that “all CPUs” may very well be most modern CPUs; RedHat advisory claims that POWER and even SystemZ (as used in IBM mainframes) may be impacted by Spectre.
https://access.redhat.com/security/vulnerabilities/speculativeexecution
Basically if your core does speculative execution for performance gain then it may be vulnerable. The BOOM RISC-V core ( https://www2.eecs.berkeley.edu/Pubs/TechRpts/2017/EECS-2017-157.html ) can do out of order execution and so _may_ be vulnerable. It would require a deeper look at other implementations of the spec to see if they are vulnerable or not.
Will this require patching the OS in our Linodes?
We’re still planning our full mitigation strategy, but Linodes will need an updated kernel, and we’re working on providing one from the Linode Manager. If you currently use a distribution-supplied or custom-compiled kernel, you will need to take separate actions to update it.
@Ruben Yes.
It appears that patching the guest is required to mitigate meltdown on Xen VM’s. If KVM, it should just be the hypervisor that needs the patches. Am I reading this correctly?
https://www.theregister.co.uk/2018/01/04/intel_amd_arm_cpu_vulnerability/
Correction, all VM’s have to be patched because of Meltdown, not just those on Xen.
Thanks Linode for your quick response and adapting 🙂
Related tweet by WikiLeaks at https://twitter.com/wikileaks/status/948723793324838914
Official website: https://meltdownattack.com
Article: https://www.theregister.co.uk/2018/01/04/intel_amd_arm_cpu_vulnerability/
A severe design flaws that allow stealing of sensitive data from memory has been discovered in Intel chipsets, affecting Xen, KVM, and more.
Linode will fix this, I trust in them. So no worries here.
Linode, can you please provide the details of patching hypervisors?
I guess mitigation of memory reads between different VMs is often even more important than within a single VM.
I am relatively new to Linode, and have a low volume site. When critical issues like this are discovered and eventually fixed, it would be awesome to get an email with directions on what to do, if anything. Thanks !!
Could you clarify how “[upgrading your VM kernel] provides us a good foundation to work with while planning for full remediation”? I mean, less attack surface is great and all, but, how does it factor into your planning? I ass-u-me you’re going to be rebooting my host at some point anyway.
It’s great to see that you are on top of it. As long as we are informed ahead of time, we are fine with the reboot and security patches. As you know our customers don’t like downtimes. Also, please try to minimize fleet-wide reboot, which makes services completely unavailable.
Thanks Linode for the updates.
I guess being on a shared (virtualized) server, all linodes on that (physical) server have to apply the new kernel, for the protection to be really effective.
That’s a good start anyway.
Thanks for updated Kernel. Looks like it doesn’t support Redhat/CentOS KPTI tunables to be able to control KPTI and related patch operations https://community.centminmod.com/posts/57936/. Would be nice to have though – details https://access.redhat.com/articles/3311301
cat /sys/kernel/debug/x86/pti_enabled
cat: /sys/kernel/debug/x86/pti_enabled: No such file or directory
cat /sys/kernel/debug/x86/ibpb_enabled
cat: /sys/kernel/debug/x86/ibpb_enabled: No such file or directory
cat /sys/kernel/debug/x86/ibrs_enabled
cat: /sys/kernel/debug/x86/ibrs_enabled: No such file or directory
were any Spectre fixes added to the kernel ? PoC at https://github.com/crozone/SpectrePoC successfully runs = not fixed on updated linode with 4.14.12-x86_64-linode92 on centos 7.4 64bit
but on dedicated elsewhere with centos 7.4 64bit and 3.10.0-693.11.6.el7.x86_64 the PoC fails to read = fixed
@George: One of the Spectre vulns requires you either recompile EVERYTHING with mitigations or a microcode patch. The other Spectre vuln isn’t fixable without newly architecture hardware (that doesn’t exist yet)
Meltdown is the one you apply the KPTI patch for (Intel only).
So Linode will probably have to issue a second round of reboots when their motherboard/OEM’s get around to issuing a CPU microcode patch. (Or they recompile everything)
Re: previous comment, ah upstream linual kernel hasn’t tackled Spectre yet according to http://kroah.com/log/blog/2018/01/06/meltdown-status/ but some distro backported kernels have i.e. Redhat/CentOS
you say “If you are leveraging a Linode kernel, upon your next reboot your Linode will be upgraded to this version.”.
Is the easiest way to tell that by doing a `uname -a` and seeing if the string contains `-linode` e.g. “4.9.50-x86_64-linode86”
@Patrick,
If `uname -a` is showing “somethingsomething-linode”, your kernel is coming from Linode. The particular version is assigned each time the instance boots, so if it is out of date, just reboot your Linode.
If you aren’t sure about the kernel source or want to change it, you can log into the Linode Manager and click the “edit” link for your Linode’s Configuration Profile. The Kernel option under Boot is where this particular setting is stored.
Some of the OS patches are requiring microcode changes. But I think the linux ones work without and differently once the microcode has changed.
The microcode will come in the form of a firmware update from the vendor.
Is there a reason why the Intel microcode update cannot be used directly rather than waiting for vendors to repackage it?
Ie, https://downloadcenter.intel.com/download/27337/Linux-Processor-Microcode-Data-File
Apparently the earlier mentioned link is not the latest version, not sure of the direct Intel link but it’s apparently included in eg https://launchpad.net/ubuntu/+source/intel-microcode/3.20171215.1
The question remains the same regardless.
excelent
Hi,
Any idea how long the reboots will take once they are scheduled?
Thanks!
Neil
We’ve allocated a two hour window for maintenance, however in many cases the actual downtime will be less. That being said, we would still recommend preparing for a full two hours of downtime.
The link given in “Jan 11 update” gives misleading information.
“Spectre targets the way modern CPUs work, regardless of speculative execution” is incorrect.
Both Spectre and Meltdown take advantage of “speculative execution”.
While Meltdown exploits a race condition based on the code after an exception is triggered, Spectre relies on the code speculatively run after a ‘if’ branch (uncached) condition that “usually” goes through happens to be false (since the next code accesses an out of bounds array).
I used to use linode vps, very good network speed.
> Our schedule runs through January 18th.
Are you kidding me.. Why so long to patch Meltdown?
Rather let your customers stay vulnerable to the exploit than lose a few customers because of insufficient server capacity?
Hey Krian!
Due to the scope of this vulnerability, we are rolling out the patch in waves to balance downtime for customers as well as ensure the patches work effectively across the entire fleet. With the hasty release of the kernel patches, we are making sure the patches don’t cause more issues for our customers than they fix.
So, my service is spread accross 5 linodes. Bringing down my VMs, one at a time, at unknown intervals spread over the next several days, is going to cause me to have possibly *five* outages in the worst case (if my VMs are all on different physical hosts, which I have no way of knowing), rather than one.
@Neil Ticktin
It took one minute to fix.
According to Uptime Robot, the monitor (my linode VM, Linode 2048) is back UP (Keyword Exists) (It was down for 0 minutes and 49 seconds).
I am relatively new to Linode, and have a low two volume site. Today I got the email with this subject “Linode Support Ticket 9678973 – Critical Maintenance for CPU Vulnerabilities (Meltdown) “. it would be great, If you provide an email what to do exactly, or anything.
Thanks !!
Hey Maneesh, first of all, welcome to Linode! For these Critical Maintenance for CPU Vulnerabilities (Meltdown) tickets, there is no action required on your end. That being said, we do recommend making sure your Linode is set to the latest kernel, which you can read more about how to do here. We would also recommend taking a look at the Reboot Survival Guide to ensure these reboots and migrations have as little impact on your Linode as possible.
I’m still seeing “Maintenance is not yet scheduled” on my dashboard.
Warning would be good it sounds like there is a schedule.
Hi Adrian! Wo don’t have a full schedule of exactly what host will experience the mitigation at what time just yet, however once we do set the schedule for a host your Linode is on you will be alerted with a ticket and via the dashboard.
How long reboots will take, once it is scheduled? Thanks in advance!
The maintenance window for hosts is 2 hours, however we expect the reboots to not take the full two hours. Beyond that, I’m afraid I can’t really give a more accurate assessment of how long the reboot will take. Hope this helps!
Oh. It is serious issue that we need to mitigate.
The whole operation, from shutdown to server back up and running took 11 minutes.
Mine reboot was around 45 minutes.. I moved my servers for now in DigitalOcean. I do not know what DO is doing but as of the moment the do not promise down time..
@Romel
That is hilarious. DO hasn’t done their reboots yet. So you leave a hosting provider for a serious, unavoidable reboot to another that has to do the same thing!
Needed that laugh this morning.
You’ve schedule 2/3 of my cluster hosts for the same window, and given me no way to reschedule that. Support has not responded to my message about preventing downtime on my cluster by either rescheduling or migrating one of my nodes to another host. That means in 11 hours my cluster will be broken.
What happened linode? You used to be good about handling these outages, but last summer you started to suck. Please improve, I’d hate to end a 4 year relationship over this.
Hey Zach, I’m terribly sorry we haven’t gotten back to your ticket yet. Can you let us know the ticket number so I can take a look and see what we can do for you? Please also feel free to call our support line for more immediate assistance as well. Thank you for being patient with us during this process!
reaaaally wish you’d had a pool of servers that we could migrate onto, on our own schedule instead of this middle-of-the-night stuff. You’ve done it before.
I apologize, due to the scope and severity of these vulnerabilities, we are unable to be as flexible as we would like with the mitigation process. If you open a ticket or call the support help line, however, we will be happy to see what we can do for you.
@Zach: Ask support for server migration for one of the cluster nodes to another hypervisor before reboot…
Hmm.
The mugration og my first linode scheduled, could be started when it fit my schedule. Why not the next? (Both nodes are located in london)
/Henning
I have a question about notifications. In the past, when downtime was scheduled for one of our systems, we’d receive an e-mail giving us plenty of advance notice. With the reboots for Meltdown/Spectre, however, we were left to polling our Linode Manager page to see what, if anything is scheduled.
From reading an article at Ars Technica, it appears that Linode learned of these vulnerabilities like the rest of us — with no advance notice. I cannot imagine the scrambling that must have caused!
As you would have appreciated advance notice, so too would we.
We’ve been hosted on Linode coming up on 4 years and currently have 9 servers here. One of the reboots took down 3 of our nodes at once. As one of these handled DNS, the lack of sufficient notice prevented us from redirecting to our redundant system (TLS propagation issues). Fortunately, that reboot happened quickly and our loss of service resulted in limited down time for our site (less than 15 minutes).
As further phases of remediation are planned, we expect that each of our systems will see at least one more reboot.
How can we get email notification of upcoming server reboots?
I completely understand where you’re coming from. For this round of reboots we aimed for at least 24 hours notice via ticket. For future reboots we’ll be able to provide notice further in advance, which will allow you more time to plan for any maintenance.
You’ll be notified of any ticket updates via the contact email address on file. If you’re not receiving notices via email let us know in a ticket and we’ll be happy to take a look.
Thank you for the prompt response!
Best wishes to you as you try and deal with rolling out fixes to thousands(?) of systems and deal with anxious customers.
We have continued to receive e-mail messages informing of system reboots… *after* they occurred. The last e-mail message *predicting* a reboot was sent on Jan 11, 2018. We had 7 systems reboot after that, and only learned of those ahead of time by constant scanning of our Linode Manager page.
This is both error-prone and time-consuming.
Our Linode Manager Notifications tab ( https://manager.linode.com/profile/notifications ) currently shows:
Linode Events Email
Events Email Notification
Notifications are currently: Enabled
So we *should* be receiving e-mail notifications in *advance* from now on? Or is there something else that must be done?
Yes, you should receive an email notification before any reboots from here on out. Email notifications should be much more timely moving forward. Even if there is a delay, the increased notice we’ll be providing for future maintenance will allow ample time for email notifications to reach you before any reboots take place.
That is a tremendous relief — Thank You!
That is one thing Linode has made a name for itself — being open and forthright in dealings with its clients. Thanks for holding your standards high!
This is unbearable. Our linodes go down with no notice. Went down twice in the last 12 hours for about an hour each time. The scheduled downtimes have been VEEERRRYYY slowish, also up to 1 hour which is unheard of in professional hosting… Lots of our services affected, our users outraged… What are u going to do about it folks??
Hello George, we are terribly sorry for the inconvenience. Due to the scope and and serious nature of the vulnerabilities, we had to perform the maintenance in an expeditious manner. You should have had tickets open informing you of the downtime in at least 24 hours notice. The maintenance window to address the vulnerability was 2 hours, however often the maintenance itself was shorter within about the hour mark. Do you currently have a ticket open so we can take a look at your account and confirm exactly what happened? I thank you for being patient through this process.
Hello,my maintenance status show phase 1 complete and future maintenance is pending,i want to know when will the migration plan come to an end。
now i try to ping my linode IP aways show request timeout, thank you!
This round of maintenance has completed, and your Linode should have returned to its previously booted state. I’m sorry to see the pings are timing out when you attempt to connect to the Linode. Do you currently have a ticket open with us to we can take a deeper look at this?
i had open a ticket to you,please help me to solve the problem,thanks!
Can you please provide the ticket number? Thank you.
Hello,my maintenance status show phase 1 complete and future maintenance is pending,i want to know when will the migration plan come to an end。
now i try to ping my linode IP aways show request timeout,
my ticket number 9869831
thank you!
We don’t currently have an ETA on the next maintenance window but tickets will be sent out once that is determined. We are currently waiting for patches to come from our hardware vendors.
We have a medium size fleet of Linodes and are seeing random loss of connectivity (requiring a reboot from Linode control panel to fix) across this fleet. This is causing major problems; last one to go was a primary database server which took out most of the fleet, and the nature of the networking problems is not playing nice with our cluster failover (eg the failures are inconsistent so the cluster might think the master it elected is fine, but outside the cluster its not visible – so the whole application is broken).
Linode support is aware of networking issues and advised us to reboot onto latest kernel versions which, unfortunately, do not seem to have resolved these issues.
Hi
I’ve set passwordauthentication NO
But now i can’t log in with my public key any more.
the public key remind the same and nothing change.
It show:
PuTTY Fatal Error
Disconnected: No supported authentication methods available (server sent: pulickey,gssapi-keyex,gssapi-with-mic)
Should I wait until the maintenance done so I can log in?
We’d like to look into this with you. Could you open a ticket for us and let us know the ticket number so we can locate it on our end?
You may be able to use Lish to log into your Linode if it’s up and running, even if SSH isn’t working:
https://www.linode.com/docs/guides/using-the-lish-console/
ticket 9859656
@paul, what would be a medium sized fleet?
Linode, can you please provide the details of patching hypervisors?
I guess mitigation of memory reads between different VMs is often even more important than within a single VM.
We just completed fleet-wide reboots for Spectre v1/v2 mitigation and are now working through a few rounds of migrations in order to close this out. More information can be found on the status page here: https://status.linode.com/incidents/8dbtk37dwm67/.
None of your servers need need BIOS/firmware updates for any of the recent CPU vulnerabilities? Would you consider providing at least some hardware with all AMD & Intel “management” features disabled? That seems like it would be 100% unique offering for a cloud host.
Also, I would like to know if you use Intel ME for any management done in the datacenters – or you use other tools (which I think are more suited to managing datacenters).
Hi Wayne – In addition to switching to our latest patched kernel (5.1.5), we are addressing these vulnerabilities at the host level during scheduled maintenance windows. This guide has additional detailed information on these vulnerabilities as well as their mitigation.
As far as providing “hardware with all AMD & Intel ‘management’ features disabled,” I have added your suggestion to our internal tracker.
Regarding your last question about using Intel ME or other tools, we aren’t able to discuss specific information like this. Though if you have any other questions, let us know and we’ll be happy to provide as much information as we’re able to.