I’m looking for just a few Linode 64 users currently on hosts in the Fremont datacenter (host9 and up) to help test a new performance patch to the host and UML.
UML works by using ptrace, a function of Linux to intercept system calls made by applications. When your UML makes a system call, the host intercepts the call, performs the work inside UML, and null-ifies the system call on the host by converting that system call to “getpid” on the host. The problem with this is that it generates two context switches on the host.
With the [url=http://perso.wanadoo.fr/laurent.vivier/UML/
]sysemu patches[/url] on the host and in UML, it doesn’t bother running “getpid”, thereby reducing number of context switches from 2 to 1.
[b]Benchmarks[/b]
Here are some “real-world” benchmarks using Mysql’s run-all-tests script:
http://www.theshore.net/~caker/uml/sysemu-benchmarks.txt
In short, the tests ran [b]33% faster[/b] than they did without the sysemu patches.
[b]Details[/b]
[list][*] We’ll migrate you to host25, keeping your slot on the previous host reserved (in case we need to migrate you back). In case you’ve never migrated before, it’s an automated process which moves your filesystem images to the new host. No changes are required on your end, and downtime is limited to however long it takes to copy your images (usually less than 15 minutes or so for 3GB).
[*] Set your config profile to boot the 2.4.26-linode29-1um kernel
[*] Try to break things[/list]
If things go well for a few days, I’ll consider it stable and open up the server to the public. After another few weeks, we can move the other hosts to this kernel.
Secondly, this will help test 2.6.7-rc1 on the host, an upgrade from the current 2.6.4 kernel.
Thirdly, I would also like for some of you to test the 2.4.26-linode29-1um kernel, to make sure it still functions correctly on hosts without the sysemu patches. Please let me know your experience with this kernel. On non-sysemu enabled hosts, it should perform the same as the previous linode kernels.
Let me know if you’re interested…
Thanks!
-Chris
Comments (17)
Three updates in one test? You’ve got ball-balls Chris!
😀
— James
I’m interested….let do it!
[quote:0999715663=”mcowger”]I’m interested….let do it![/quote]
Great. I haven’t patched a 2.6-um kernel with the sysemu patches yet, so for now go with the 2.4.26-linode29-1um kernel. I’ll update this thread when the 2.6 version is out.
Thanks!
-Chris
I’m game. What do I need to do?
Terry
[quote:31f75d1a84=”terryr”]I’m game. What do I need to do?[/quote]
Login, shutdown, and press the migrate button. After the migration, reboot.
BTW, 5 users have moved over to host25 and so far so good.
-Chris
Done! Are you looking for any specific feedback?
[quote:5c2da2598a=”terryr”]Done! Are you looking for any specific feedback?[/quote]
I pretty much know there’s a performance improvement already, but hearing about that would be nice. It might be kind of difficult to pin down, unless you run specific tests and compare them against the linode28 kernel. Secondly, I wanted to generate some load on the host kernel, and the linode29 kernel, just to make sure the stability is still there.
Thanks!
-Chris
I loaded a fresh Fedora Core 1 distro. Noticed immediately how much faster it is. Ran locate -u which was noticeably faster. Ran top and saw something strange. About every two seconds, there’s a cycle. See pics below. irq goes up to 0.9% and softirq goes up to 49.5% and iowait is 49.5%, then it all drops to 0 for 2 seconds and goes back up again for two seconds.
[img]http://futurecourse.com/files/no_softirq.gif[/img]
[img]http://futurecourse.com/files/softirq.gif[/img]
I don’t recall this happening when I looked at top on the old host. Or did I just miss it?
Terry
I am seeing stragne problems with mysql that I did not see before moving over. Whenever, I try to connect I get this message:
ERROR 2002: Can’t connect to local MySQL server through socket ‘/var/run/mysqld/mysqld.sock’ (2)
This is because mysqld either keeps hanging or completely crashes.
It will run fine for a few hours and then just die. I can’t explain why.
I will keep everyone posted.
John
[quote:04e5122a27=”jftuga”]I am seeing stragne problems with mysql that I did not see before moving over. Whenever, I try to connect I get this message:
ERROR 2002: Can’t connect to local MySQL server through socket ‘/var/run/mysqld/mysqld.sock’ (2)
This is because mysqld either keeps hanging or completely crashes.
It will run fine for a few hours and then just die. I can’t explain why.[/quote]
I’ve noticed your Linode using a lot of swap, so perhaps you’re running out of memory and the OOM (Out Of Memory) killer is kicking in…
-Chris
[quote:2ab259ee1d=”terryr”]I don’t recall this happening when I looked at top on the old host. Or did I just miss it? [/quote]
I’ll give this a shot, too, but.. would you mind rebooting into the linode28 kernel and seeing if you can reproduce this?
I somehow doubt the SYSEMU patch would be the cause of this, or the mysql problem…
-Chris
I have modified the mysql server for a small memory footprint using the my-small.cnf that comes with the src distro.
I will let you kno if this helps.
Also, I am using a lot of swap, and am considering upgrading to a 96 or 128 when they become available.
-John
[quote:befbb85d95=”caker”]I’ll give this a shot, too, but.. would you mind rebooting into the linode28 kernel and seeing if you can reproduce this? [/quote]
Did that yesterday with no change, but when I checked this morning, I’m at 0.1% for irq and softirq and it cycles with iowait at 0.2%. The high initial numbers didn’t seem to affect performance at all.
—
Terry
where is the migrate button? I shutodown and clicked around but I never found it.
[quote:f84ba42be4=”bakins”]where is the migrate button? I shutodown and clicked around but I never found it.[/quote]
This is something I have to set up for you first (which I just did). Login and you should see it.
Testing has gone very well. In fact, the machine is “live”, 2.6.7-rc1 is performing great (on the host), and the linode29 kernel works perfectly on SYSEMU and non-SYSEMU patched hosts.
I consider this test done, but go ahead and make the move — let me know what you think.
-Chris
I was out of town for a week.
I’ve now migrated and am testing. Everything seems to work. I’m testing out plone. I’ll do some tests and get back.
interesting… is this similar to the skas patch?