Today I found a really good working plug-in for Pidgin which allows you to use the most Skype features directly from Pidgin: http://code.google.com/p/skype4pidgin/

You still need Skype to be running, but you got all you contacts in you Pidgin list. If you make a call the standard Skype dialog appears, all message communication is done via the usual Pidgin conversation windows.

Skype 4 Pidgin

Skype 4 Pidgin

Downloading

I used a small python script youtube-dl for downloading the video data.

• It is available in the Ubuntu repositories

  sudo apt-get install youtube-dl

• If you encounter some problems you also could do a source checkout from BitBucket (requires that Mercurial is installed):

  hg clone http://bitbucket.org/rg3/youtube-dl/

• Download the video from a terminal:

   youtube-dl http://www.tube.com/watch?v=videoid

• After the download you will find a videoid.flv in your folder

Getting the audio data

For extracting the audio data I used ffmpeg.

• If you haven’t installed it already:

  sudo apt-get install ffmpeg

• We will extract the audio data to a simple wave file. This file is gonna be large, so make sure you delete it afterwards.

  ffmpeg -i videoid.flv output.wav

Converting to MP3

For converting the wav file to a mp3 file I used lame.

• Run (the “-b” arguments sets the bitrate of the mp3):

  lame -b 192 -h output.wav output.mp3

Notes

• There a lot of other ways to get the data. Instead of lame you could use mplayer, but this way I alway ended with broken mp3 files.
• Instead of converting to wave file you also could create a mpeg video and then create the mp3 file directly from this video.
• Lame and also ffmpeg have tons of options available from the command line, which could increase speed and quality of the conversation.

The offline alternative

A looked around for some alternatives and the first found was the MySQL Query Browser which is part of the MySQL GUI Tools. It is fast (okay, it is a installed application, but still its fast) and very comfortable for editing and creating tables and databases and also okay for just filling / editing some data inside the tables. Also nice is that is free.

MySQL Query Browser - result view

MySQL Query Browser - Table option view

In the MySQL GUI Tools there is also a so called MySQL Administrator which is nice for some statistics, setting up users and creating and restoring backups.

The online alternative

There is only one but a big con with the MySQL Query browser. I can not run it on my server, because therefore I needed to open the MySQL port and that one thing I definitely will not do.

So I searched for a while on the web and the tool which looked best for me was SQL Buddy:

• PHP-based
• makes heavy use of AJAX
• enough options for the most common operations
• size just 1 MB (instead of 11 MB for the full version of PHPMyAdmin)

SQL Buddy - result view

SQL Buddy - Table option view

For the most common tasks SQL Buddy is perfect. It is nearly as fast as installed application and easy to use.

But nobody is perfect, there are few thing I found out, that could not be archived with SQL Buddy. There is no option to create and import xx.sql.bz2 files, and currently no handling of the different storage engines, which means that you could not use foreign keys for example.

Conclusion

For you local computer you are okay with the MySQL GUI Tools and SQL Buddy, on the server I will use SQL Buddy but for the more complex tasks I will stick to PHPMyAdmin.

I make no secret of that I am a big fan of the Firefox web browser. Except of a few things that I will name later I am/was always impressed by the speed improvements that are made over the past few years, which seems very important to me, because many pages you use in you daily life are more and more based on heavy JavaScript functionalities. A few days ago I decided to give a try to the newest development version of the Firefox – named Minefield – and make some tests.

Building

Instead of using a nightly build I decided to to compile it by myself, which was not that hard. If you like to do it yourself could simply follow the instruction on that page: Simple Firefox build. I used slightly modified the mozconfig file in hope to get the best performance:

. $topsrcdir/browser/config/mozconfig
mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/objdir-ff-release
mk_add_options MOZ_MAKE_FLAGS="-j4"
ac_add_options --disable-tests
ac add_options --disable-crashreporter

Testing

As performance test I used the Sunspider JavaScript Benchmark from the Webkit developers. To make it short, here are the result for 3 versions of Firefox:

a

time in milliseconds, faster is better

time in milliseconds, faster is better

Results

When I switched from from Firefox 3.0.x to the 3.5 branch of Firefox I was really amazed by the performance boost gained through the new JavaScript engine Tracemonkey. As the results above show the next Firefox releases will get another boost. Good news!

P.S.: Other things I like Firefox for is because of all the very useful add-ons, I hardly could imagine a day as a web-developer without the use of Firebug. AdBlock saves my nerves every day. Linkification, TinyURL generator and Greasemonkey are just a few other I really like and needed to be named here.

So how to convert this gif animation into a swf file, which the Flash Player could play? (And of course I only want to use open source tools.) It was very easy:

• Make sure you have ImageMagick installed.
• Open a console and go to the directory where the gif file is located:

  convert +adjoin -coalesce loading.gif frame%02d.png

• Now we have a png for every frame of the gif animation. Why png? Because swfmill does not support gif as file format and png it the only available format with support for transparency.
• Create a XML file that look similar to this one:
  1. <?xml version="1.0" encoding="utf-8" ?>
  2. <movie width="32" height="32" framerate="25">
  3.         <background color="#ffffff"/>
  4.         <frame>
  5.                 <library>
  6.                         <clip id="frame00" import="frame00.png"/>
  7.                 </library>
  8.                 <place id="frame00" depth="1"/>
  9.         </frame>
  10.         <frame>
  11.                 <library>
  12.                         <clip id="frame01" import="frame01.png"/>
  13.                 </library>
  14.                 <place id="frame01" depth="1"/>
  15.         </frame>
  16.         <frame>
  17.                 <library>
  18.                         <clip id="frame02" import="frame02.png"/>
  19.                 </library>
  20.                 <place id="frame02" depth="1"/>
  21.         </frame>
  22. ….
  23. </movie>
• Finally merge the single frame images together with swfmill:

  swfmill simple input.xml output.swf

• That’s all.

HTTP/1.1 200 OK
Date: Sun, 06 Sep 2009 11:44:56 GMT
Server: Apache/2.2.8 (Ubuntu) DAV/2 SVN/1.4.6
X-Powered-By: PHP/5.2.4-2ubuntu5.7
Connection: close
Content-Type: text/html

Actually I think no one needs to know which PHP version is running on this machine and that there also a Subversion is running on that machine. Even the name of the operating system and the version of the Apache is not needed in most cases.

So how suppress these information?

Disable the “X-Powered-By”

• Log in to your server as root user.
• Open you php.ini file in a editor of you choice. My php.ini is located at /etc/php5/cgi/ (as I use mod_fcgid to server PHP content)
• Search and edit:

  ; Decides whether PHP may expose the fact that it is installed on the server
  ; (e.g. by adding its signature to the Web server header).  It is no security
  ; threat in any way, but it makes it possible to determine whether you use PHP
  ; on your server or not.
  expose_php = Off

  If you like you read in the official PHP documentation.

• Save and reload you Apache:

  /etc/init.d/apache2 force-reload

Disable the “Server”-header

After searching for while I figured out that disabling the “Server”-header is not possible, without recompiling the Apache. (The Apache developers claim this as a feature.)  But you could decide between some level of information that will be provided.

• If you are not logged in, log in as root user.
• Open you Apache configuration, for me it was /etc/apache2/apache2.conf
• Search for and edit the following lines:
  1. # ServerTokens
  2. # This directive configures what you return as the Server HTTP response
  3. # Header. The default is ‘Full’ which sends information about the OS-Type
  4. # and compiled in modules.
  5. # Set to one of:  Full | OS | Minor | Minimal | Major | Prod
  6. # where Full conveys the most information, and Prod the least.
  7. #
  8. ServerTokens Full
• Replace the Full with some of the options given in the comment. Here are some examples what these options are meaning:
  

  Full	Server: Apache/2.2.8 (Ubuntu) DAV/2 SVN/1.4.6
  OS	Server: Apache/2.2.8 (Ubuntu)
  Minor	Server: Apache/2.2.8
  Minimal	Server: Apache/2.2
  Major	Server: Apache/2
  Prod	Server: Apache

  If you like you could also read the official documentation.

• Save and reload you Apache:

  /etc/init.d/apache2 force-reload

Result

I had chosen Major in the last step. Now the headers look like this:

HTTP/1.1 200 OK
Date: Sun, 06 Sep 2009 12:16:40 GMT
Server: Apache/2
Connection: close
Content-Type: text/html

Important note: This does not improve the security of the server, because you are only hiding information. Maybe you want to read: Security through obscurity.

It seems that keeping up the the apache mpm_prefork and mod_php will not be the solution anymore. Most sites suggested to use mpm_worker instead, because it uses less memory (and my problem was, that my Apache consumed all my memory). On the other hand there could be some problems with thread-safety, but I liked to give it a try. PHP will not me used as an Apache module any more. Instead we will run PHP as a FastCGI script.

A short step by step tutorial for switching from mpm_prefork to mpm_worker with using PHP.

• Open a terminal session as a root user.

  apt-get install apache2-mpm-worker libapache2-mod-fcgid php5-cgi

  apt will complain about a few things and will remove some of the previous installed packages. After installation make sure that mod_fcgid is enabled.

• Edit you /etc/apache2/apache2.conf or /etc/apache2/httpd.conf or any config file that will be parsed by apache and add this:
  1. AddHandler fcgid-script .php
  2. FCGIWrapper /usr/lib/cgi-bin/php5 .php
  3. Options ExecCGI FollowSymlinks Indexes
• Now enter you /etc/apache2/sites-available directory and edit all the the files there. Every file should contain at least one “VirtualHost” node. Within these “VirtualHost” nodes there should be one or two “Directory” nodes. If not already there add a new line starting with “Options” and add “Indexes” and “ExecCGI” as parameters on these lines. It should be look like this one:
  1. Options Indexes FollowSymLinks MultiViews ExecCGI
  2. AllowOverride AuthConfig
  3. Order allow,deny
  4. allow from all
• Restart apache:

  /etc/init.d/apache2 restart

Hopefully now everything should work as before. If you had enabled eAccelerator or any other byte code cache system or if you had mode some modifications on you php.ini you now have to copy them to your “new” php.ini located at /etc/php5/cgi/php.ini

A few things wont work as before, e.g. when use user authentication by PHP and not by htaccess. There are also a few issues I read about, but nothing which affect my sites.

As a result I would say spending this hours messing around with Apache and mod_fcgid was a good investment. My server now need only 90 MB Ram when idle, it was 250 MB Ram before. Even under load memory consumption stays lower as before. I hope this will be the last tweak for I longer time.

Using PHP as a FastCGI script gives you a lot of new options. So if you have a lot of time to spent or very worried about the security of you web server you could use the Apache module suexec which then could execute the script as a special user and not as the default web server user www-data. It could even use multiple user for each virtual host / directory.

Interesting links:

• http://brian.moonspot.net/2008/02/13/apache-worker-and-php/
• http://www.ibm.com/developerworks/linux/library/l-tune-lamp-2.html Tuning Apache and PHP
• http://www.ibm.com/developerworks/library/l-tune-lamp-3.html Tuning MySQL
• http://ivan.gudangbaca.com/installing_apache2_and_php5_using_mod_fcgid More detailed instruction how to install mpm_worker and PHP via mod_fcgid

• 20 GB Raid 5 Storage
• 512 MB RAM, max 2 GB RAM
• 2 TB Traffic

At this time I thought it would be nice to install every thing by myself, so I choose a plain “Ubuntu 8.04 LTS 64 bit minimal” as operating system for my vHost. I installed Apache2, PHP5, MySQL and all the stuff I needed. It all works fine and I was really impressed by the speed of this virtual server.

Then I moved sportinleipzig.de, which I made during the time at my last company to this new server. This page does not have a very high traffic (200 unique visitors a day), but a lot of content (at least 5000 “static” pages). After 2 days the server crashed the first time, whereas crash means, that he is still alive, but does not respond to anything more then a simple ping. I restarted the server over the admin panel. Half a week later the server crashed the next time. Investigating the cause for these crashed I found out a few interesting things.

First a blamed mod_tora which I installed on this virtual server to check if it is ready to use for some production code. Then I deactivated eAccelerator, because I thought it could be possible that it has some memory leaks. I even deactivated Piwik which I use as a tracking system for all the visitors on the pages of this server. (It is very nice! If you need some visitor tracking, give it a try.) But the server still crashed ever few days.

Later I recognized that these crashes possibly occur when Google starts to crawl the page, as the page was not completely indexed anymore. So I started to do some load test on the server. A good page to start is loadimpact.com and the Apache command line tool ab, which both give you the possibility to run a lot of requests on you own server. Having a ssh session open and refreshing the status page of the Virtuozzo panel it became more and more clear. These kind of virtual server are very restricted how many resources they could consume and there a few limit that are easy to reach:

• kmemsize:
  

  The size of unswappable kernel memory allocated for the internal kernel structures for the processes of a particular VPS.

• shmpages:
  

  The total size of shared memory (including IPC, shared anonymous mappings and tmpfs objects) allocated by the processes of a particular VPS, in pages.

• numproc (maximum set to 128!):
  

  The maximal number of processes the VPS may create.

• numfile (maximum set to 8192, this is okay):
  

  The number of files opened by all VPS processes.

• numtcpsock and numothersock (both set to 720, means: no socket servers on virtual servers!):
  

  he number of TCP sockets (PF_INET family, SOCK_STREAM type). This parameter limits the number of TCP connections and, thus, the number of clients the server application can handle in parallel.

  The number of sockets other than TCP ones. Local (UNIX-domain) sockets are used for communications inside the system. UDP sockets are used, for example, for Domain Name Service (DNS) queries. UDP and other sockets may also be used in some very specialized applications (SNMP agents and others).

There are a lot of other limits set to the server, but the first three are these limits that made my server crash, when having tons of requests on the Apache.

I am using the Apache2 in the prefork mode (this is default if you install it via the Ubuntu repositories). This means the Apache spawns a few processes to handle the requests. If load raises more processes will be created, if load goes down unused processes will be destroyed. Forking new processes cost a little bit time so the Apache keeps some processes ready just in the case of load peak. During the time google crawls the page many processes where created to handle the load. The problem was that these processes consumed to much memory (probably this could be a memory leak in one of the PHP  scripts) and some of some became zombies, leading the virtual server to consume all resources and then crash.

The solution:
Edit you “/etc/apache2/apache2.conf”. Search for this section:

You could read in the comment of the config the impact of every parameter. The important one is the last one which is set to “0″ by default, which means that processes live as long as possible. Setting it to “500″, which is quite conservative, makes every process ends after serving 500 request. This also makes sure that all the allocated memory is freed. That causes that processes need to be created more often, so I also raised “MinSpareServers” and “MaxSpareServers”, which are the processes keept ready for load peak.
In conclusion my server spends a little more time on spawning new Apache processes, but this makes sure memory is cleaned more often and finally solved my problem!

P.S.: I don’t know if this is on all virtual servers, but the vHosts at 1und1 have a “/proc/”-entry for reading the limits of you server. You type into the terminal:

root@foo:~# cat /proc/user_beancounters

PHP is not the fastest scripting language, that truly a fact. Except from optimizing the scripts there a few possibilities to speed everything up. One is to install a byte code optimizer and cache on your server. They cache the byte code created by the PHP parser and try to optimise it. So every time some one requests for a page on your server the script does not have to be parsed again. This brings a speedup of 120 – 250% depending on you script. There are several byte code caching program available, the one I like most is eAccelerator. Sadly eAccelerator is not available through the Ubuntu repositories so you have install them yourself. Here is a short description how to archive this!
Everything is done from the terminal as a root user.

• Prepare you system for compiling a PHP extension:

  apt-get install build-essential php5-dev

• Choosing a folder for build process. I used “/usr/local/src/”
• Getting the sources from the eAccelerator site:

  wget http://bart.eaccelerator.net/source/0.9.5.3/eaccelerator-0.9.5.3.zip

• Extract and delete the archive

  unzip eaccelerator-0.9.5.3.zip && rm eaccelerator-0.9.5.3.zip

• Enter the folder

  cd eaccelerator-0.9.5.3

• Prepare as a new PHP extension and compilation

  phpize
  ./configure --enable-eaccelerator=shared

• Compile and install

  make
  make install

  The install process prints out a folder where eAccelerator has been installed. Remember or copy it, you will need this later.

• Enabling eAcceletor: Editing your “php.ini” or create a new ini file in you conf.d directory of PHP. (You can find them here: “/etc/php5/”.) Insert the following:

  zend_extension                  = "/usr/lib/php5/20060613+lfs/eaccelerator.so"
  eaccelerator.shm_size           = "0"
  eaccelerator.cache_dir          = "/var/cache/eaccelerator"
  eaccelerator.enable             = "1"
  eaccelerator.optimizer          = "1"
  eaccelerator.check_mtime        = "1"
  eaccelerator.debug              = "0"
  eaccelerator.filter             = ""
  eaccelerator.shm_max            = "0"
  eaccelerator.shm_ttl            = "0"
  eaccelerator.shm_prune_period   = "0"
  eaccelerator.shm_only           = "0"
  eaccelerator.compress           = "1"
  eaccelerator.compress_level     = "7"
  eaccelerator.allowed_admin_path = "/var/www/eaccelerator"

  In the first line enter the path printed out during the install. In the last line you could enter a directory for the eAccelerator admin tool, if you want to use it.

• Create the cache directory:

  mkdir /var/cache/eaccelerator
  chown www-data.www-data /var/cache/eaccelerator

• Restart you Apache!

  /etc/init.d/apache2 restart

There are a lot of setting in the ini file. The whole list and the description of them could be found here:  Ini-settings.

To check if everything is working open up a page on the server that contains the “php_info();” command. There should be some thing like this:

eAccelerator

Enjoy the speed!

After a while I decided to start a blog again and here is my first post.  (I think this is the third try, so we will see how long this experiment will  last.) Most of the blog entries will be about typical nerd and computer stuff like linux servers, programming, etc. I hope there will be some interesting topics for you.