Downtime

A few days ago I had an interesting case of a server downtime. The server is just a playground for developers, so no big deal. But still, lessons learned.

The reports came almost simultaneously from developers and from the monitoring system, "cannot connect". And indeed, the server was not pingable. Someone else's server, with IP equal to the IP of our server with the last octet increased by 2, was pingable, so I concluded it was not a network problem.

Next reaction: look at the server's screen, using remote KVM provided by the hoster. Kernel panic! OK, need to screenshot it (done) and reboot the server. Except that the Power Control submenu in the viewer is grayed out, so I can't. And a few months ago, when we needed a similar kind of reset, it was there.

OK, so I created a ticket for resetting the server manually. And I had to remind them that the remote reboot functionality is supposed to work. Here is the hoster's reply (PDU = power distribution unit):

Dear Alexander,

Upon checking on the PDU, the PDU is refusing connection.

We'll arrange a PDU replacement the soonest possible.

We apologise for the inconvenience caused.

Everybody reading this post, now, please check that you don't fall into the same trap. Run your iKVM viewer against each of your server that it can connect to, and check that it runs, and that the menu item to reset the server still exists. Create a calendar reminder to periodically recheck it.

And maybe append "panic=10" to your linux kernel command line, so that manual intervention is not needed next time.

Recovering a bricked D-Link router

Today my D-Link DSL-2600U ADSL/Wi-Fi router decided to die, without any obvious reason. Only the power and Ethernet LEDs were active, and the computer directly connected to it via Ethernet could not receive an IP address. Fortunately, I had an old ADSL router still around, and thus was able to find information that, in the end, allowed me to recover the DSL-2600U.

The first thing I thought, however, was that a problem is on my local computer. So I started tcpdump, only to find out that the router sends strange packets: a gratious ARP for 192.168.1.1, followed by ARP "Who has 192.168.1.159? Tell 192.168.1.1". It didn't respond to pings even from 192.168.1.2. At this point, it became obvious that it is broken or at least needs its settings to be reset, without any access to the web interface. Pressing the Reset button in the pinhole did not change anything.

When assigning my computer the IP 192.168.1.159, the conversation went a bit further, namely, there appeared a TFTP packet with a string "308nfs6358" in it. Googling for it (via the spare router) gave mostly results about reflashing that same router model that I had, via the COM port, as a last resort. They were useless, as adding a COM port to the router is not a modification that I wanted to do.

Still, I wanted to try to revive the router, and these pages also suggested that TFTP is involved somehow. They even told the reader to put the official firmware to 192.168.1.1 via TFTP after pinging it - but obviously this could not work at this stage.

So, as the router asked for the "308nfs6358" file, and also the I had to provide it via TFTP. On Linux, there is a tftp server, atftpd, so I set it up. It is actually easy. Just create the /tftpboot directory and chmod it to 777, then run:

atftpd --no-fork --daemon

However, it was not an obvious question what to put in that directory. I had a firmware file, downloaded from the official site. When unzipped, this yields DSL-2600U_3.12_6338_12_RU_1.25_20101129.img. However, the web pages containing the reference to the "308nfs6358" file also mentioned "bcm963xx_fs_kernel", so I assumed (maybe wrongly) that I had to split the original firmware file into two.

This OpenWRT page ended up being a useful resource. It contains a program to dump the firmware structure. As the program calculates and verifies CRC of various portions of the firmware file, I guessed that I could modify it to dump these portions. The implementation is obvious: there is a compute_crc32() function that receives the offset and length of the region being checksummed, and it is trivial to dump the bytes into a separate numbered file. So, I ended up with a short 256-byte header and two files. For the reference, here are the offsets and lengths:

rootfs: start=256, length=3063808
kernel: start=3064064, length=560856

As the router was broken anyway, and because I had no better idea, I decided to determine the correct file by trial and error. To be able to understand what happens, I started Wireshark and told it to sniff all packets on the eth0 interface of my computer. I copied one of the files (the shorter one) into /tftpboot/308nfs6358 and power-cycled the router. It downloaded the file via TFTP (as evidenced by Wireshark), and then nothing happened.

With the other file, I had better luck: the router not only downloaded it, but also changed its MAC address, sent some gratious ARP packets and became pingable for some time. Still, it didn't attempt to download anything else. OK, so it means that it waits for me to upload the second file to it, I thought. This didn't work. The correct thing to do was to upload the original unzipped firmware.

So here is the complete sequence of actions that worked:

1. Assign the IP address 192.168.1.159 to the computer.
2. Unzip the firmware, split it.
3. Copy the rootfs (the 3.6 MB part of the firmware) to /tftpboot/308nfs6358
4. Run atftpd as follows: atftpd --no-fork --daemon
5. Power-cycle the router
6. When it changes the MAC address, upload the firmware to it: atftp --no-source-port-checking -p -l  DSL-2600U_3.12_6338_12_RU_1.25_20101129.img -r bcm963xx_fs_kernel 192.168.1.1
7. Try again from the beginning, as the default timeouts are rather short.

This blog post has been sent through the revived router. Still, I guess it won't last long, and I need to buy a replacement.

New TV Set

Recently I bougth a new TV set - a Sony KDL-32W5500 LCD. It replaced the old Soviet-era CRT-based "Akari" TV.

The new TV set receives both analog (still the norm in Russia) and digital stations. It contains an AVC tuner and thus doesn't need a set-top box or a separate decoder to watch the digital broadcasts (Russia, unlike the rest of the world, uses MPEG-4 Part 10, aka H.264, for them).

The default color settings are good, unlike those in the old TV, which gave oversaturated colors by default. The new TV set has a 1920x1080 panel, but I found that, for my eyes, 1080i and 720p (from the computer monitor) look nearly the same. And there is not a lot of HD content in Yekaterinburg anyway.

The default option for aspect ratio, however, is to stretch 4:3 content to 16:9 non-uniformly. The options to preserve the original 4:3 aspect and to stretch the letterboxed content uniformly do exist. I wish it could autodetect the letterbox - this would mean one less button to use.

While digital broadcasts are displayed perfectly, the analog tuner is not so good as in the old TV. Stations that were displayed with "snow" by the old TV now are not watchable at all due to either flashing color stripes or the "no signal" message.

As far as the inputs go, the Sony TV has 4 HDMI sockets, one VGA, one USB and one Ethernet connection. Using a DVI -> HDMI cable and a separate analog audio cable, this TV can be connected to my computer's video card and sound card. The intel driver had no problems recognizing it as a 1920x1080@60p panel.

However, as the primary TV show consumer is my mother, this is still not a very viable solution. I want to work while she watches films. So, I tried using the built-in media playback capabilities of this TV set.

First, it can play files from USB flash drives. The instruction says it accepts only MPEG-1, but de-facto, it accepted non-interlaced MPEG-2, too. It could not play a DVD rip from the flash drive. So, using this option means transcoding everything - not very good.

Second, it can act as a UPnP Renderer. The instruction says that the TV accepts MPEG-1, MPEG-2 and AVCHD formats. So, I thought I could set up a media server on my computer. I have tried MediaTomb and MiniDLNA. MediaTomb did not work at all (the TV displayed the "server is unsupported" message), MiniDLNA worked somewhat. I was able to send MPEG-1 and MPEG-2 files (including the DVD rip that failed to play from the flash drive) to the TV. I could not remux an available H.264 + AC3 Matroska file to MPEG-TS in such a way that the TV understands it. So, for typical torrented video files, this still means transcoding. So, I must conclude that the media playback capabilities of this TV are there only for the marketing department to be able to say that they exist.

So, I am a bit dissatisfied with this offer from Sony.

Getting more quality from onboard sound

Onboard high-definition audio chips are very common now. From the viewpoint of software, they support unbelievable quality of sound reproduction: 24 bits of precision at the 192 kHz sampling rate. The reality is in fact worse, as there is always noise in the analog output that the headphones are connected to.

I am not talking about the thermal noise or 1/f noise that is created intrinsically by every electronic device and sounds like a soft "shshsh...". These kinds of noise are important in radio receivers or tape recorders that have a high-gain amplifier needed to recover a weak incoming signal, but not in computers.

I am talking about the interference, where the signal in one wire (not intended to be played as sound) propagates into the other wire located nearby or to another circuit connected to the same badly-filtered power supply. The exact sound of such interference depends on what the computer is doing. If the computer is doing something periodically (e.g., drawing a scene on the screen 60 times a second, or filling a sound buffer in JACK 500 times a second), this unwanted sound becomes a tone, and thus becomes easily noticeable. Here are the steps that helped me to reduce it in my desktop computer based on the Intel DG965SS motherboard.

First, the wire that picks up a lot of interference is the wire that connects the front headphone socket to the motherboard. It is better to avoid using the front socket at all, and plug the headphones (if the cord is long enough) into the green socket on the back of the motherboard. This way, the cord is screened from the noisy components inside the computer by the metallic case.

Second, it may be a good idea to reduce the time variations of the power consumption of various components in the computer.

E.g., the processor draws a lot of power when it is busy, but consumes less power when it is idle. When it switches between the busy and idle states periodically, it creates a tone in the power line, and that tone ultimately gets to the sound chip output. So, to suppress the tone, one solution would be to keep the processor always busy. This is what the "idle=poll" Linux kernel parameter does. Note that a busy processor runs hot (in my computer, at 47.0°C), so think twice before using this option. A less dragonian solution would be to pass the "max_cstate=3" parameter to the "processor" module. (A big "thanks" goes to Robert Hancock and Sitsofe Wheeler who suggested these parameters.)

The other component of the computer that periodically draws a lot of power is the video card. But it is easier to keep it mostly idle, not busy. The most common desktop process that keeps the GPU busy is a compositing window manager (kwin or compiz). So, disable desktop effects, and enjoy better sound quality.