GenerationIP

Just for you

  • Increase font size
  • Default font size
  • Decrease font size
Home News Feeds Latest news from planet Centos
Newsfeeds
Planet CentOS


  • DRBD 8.3.12 for CentOS-5 in testing
    The ELRepo Project has DRBD packages for CentOS-5 and CentOS-6, named drbd83-utils or drbd84-utils.  The CentOS Project does not want to maintain extra packages that exist in other places unless we need to change them ... so we are not going to create DRBD packages for CentOS-6.

    Since CentOS-4 is being EOL'ed in less than a month, we are also not going to publish updates for the DRBD in CentOS-4.

    This leaves the DRBD for CentOS-5 that are part of CentOS Extras.  Since these have been released for CentOS-5, we will continue to maintain the DRBD version 8.3.x  tree (drbd83) in CentOS Extras.

    A new version of DRBD 8.3 (drbd83-8.3.12) has been released to the testing repository for CentOS-5.  You can see the details here:

    DRBD 8.3.12 for CentOS-5

    If you want to use DRBD 8.4.x for CentOS-5, rather than releasing it separately, the CentOS Project recommends that you use drbd84-utils from ELRepo (linked above).

    For users who want to use the drbd83-8.3.12 version ... please test the version that is currently in CentOS Testing and provide feedback.  With enough feedback I will move the packages from testing to CentOS Extras.


  • Communities and Questions

    I am often surprised by the sort of questions asked in the forums or on irc around open source projects - it just feels as if people are going out of their way to inflict pain and suffering upon themselves by trying to find the most awkward and most complicated way to do things. So how can we better help these people ? We dont need to save them or anything as drastic like that, its just a case of being able to show or explain that there might be a better way.

    The first thing that I've started now doing, when asked a strange question is ask the person 'What are you really trying to achieve?'. You might be amazed how many times the answer has nothing to do with the question being asked. Try to establish what the end goal is, and in many cases its clear that the person has been lead astray by random posts on the internet, some of which are perfectly fine in their own context, but can be quite a kludge outside that context.

    Establishing, clearly what the goal is before advice or opinion is thrown at people will always result in a better overall experience. And to the people spending their time in the irc channels, web forums and mailing lists helping others out : must respect. You guys are the ones making the idea of Communities and Open Source work.

    - KB



  • CentOS Automated QA explained …

    While Johnny was explaining to the rest of the world how CentOS 6.1 and 6.2 were released, I received quite some questions about the QA tests and how they were performed. Well, let me explain in some words how it's now organized. Previously, there was only a Tests Matrix that was shared between the QA team members : each member of that group had access to the QA bits, could download/rsync the complete tree (with ISO images too) and do his tests, and then reported the results in one way or the other (irc, mailing-list). Of course it didn't scale out very well. Too much manual intervention, and when someone was busy with personal (or work related) issues, no feedback was coming back to the CentOS devteam.

    So during Fosdem 2011, I had a meeting with Karanbir to see how we could solve that issue and put automation in the QA loop. We dedicated some (old) machines to be used only for QA, and in a separate VLAN. Basically, here are the steps from the built bits to the QA reports.

    • The CentOS buildfarm (using the newly build system called 'reimzul' and using beanstalkd as a queuing system) pushes automatically each new tree to the dedicated QA hardware
    • There is a rsync post-xfer script that is launched from there that also uses beanstalkd and some workers (so we can scale out easily if we add machines)
    • Each built and pushed tree/ISOs set has its own BuildTag (that is used to identify what was tested and when)
    • Some tools (hosted in an internal Git repository) are then used to deploy some Virtual Machines (actually a mix of BareMetal and VMs : blade/Virtual Box/Xen/KVM) and send a report if the "deploy VM step" failed (VMs are installed through ISO/pxe boot/virt-install through http/ftp/nfs methods)
    • A test suite (that we call the t_functional stack) is then copied from the local git repo to those newly deployed machines and each test is then ran. From that point a report is then automatically sent to the QA mailing-list so that people can see the results, while the full log is available on QA head node.

    The fact that we use two separate git repositories (one for the deploy/provisioniong functions and another one for the tests themselves) was really a good thing, as it permitted some people to include their tests in the t_functional stack. For example , Athmane did a great job writing/fixing some tests used for 6.1 and 6.2.

    More informations to come later about how you (yes, *you*) can participate and contribute such CentOS QA auto-tests !



  • SSD performance tips for RHEL6 and Fedora
    Solid State drives provide a pretty substantial performance boost over traditional hard drive technology, but they have some limitations that require some additional planning. There are basically two big things to do, enable discard/trim support in the filesystem, and limit write operations to the SSD. You want to enable discard to deal with underlying drive specific performance degradation

  • Oracle RAC on RHEL6 and CentOS 6
    Oracle seems to be dragging its feet in certifying the enterprise linux 6 line (RHEL, CentOS, and similar) for use with its flagship database products, but that doesn't appear to be stopping the smart folks over at dell from putting together directions for getting it working. Adam M has put together a nice piece on making Oracle and RAC work with RHEL 6, and done all the hard work for you. Have

  • CentOS in 2012
    The first thing I want to do is congratulate Karanbir and Tasha on the birth of their new baby girl Millie. She is the quite cute ... hello Millie :)

    The CentOS Project has spent much time and effort into getting a new build system in place for CentOS 6 that can generate good and timely builds, as well as inform us of newly released upstream SRPMS and keep the CentOS QA team informed when we build any new packages.

    The release of CentOS-6.2 on 12/20/2011, in less than 2 weeks and at the same time as Oracle's OEL as noted on Distrowatch, is where we would like to have all our future releases be. I think that we should see the standard 2-4 week time frame for point releases and within 24 hours for updates now that we have this new build system in place.

    We have also put a Continuous Release (CR) repository in place for both CentOS 5 and CentOS 6. This repository can be installed via the simple command:

    yum install centos-release-cr

    The purpose of the CR repository is to allow the CentOS Project to push some of the security updates if we are having issues with a point release build (like we did with both CentOS-6.0 and CentOS-6.1). If we are not going to meet the 2-4 week goal for our point release, we will push out the packages we have gotten to build properly while continuing to work on the problem packages. This repository is totally optional and was not needed with CentOS-6.2, but we recommend it be installed if you want to get your security updates as fast as possible.

    Karanbir gets the credit for the new build system, called reimzul. It uses beanstalkd work queues and allows adding new builders to process the work as required.

    The build system has the flexibility to allow us to import SRPMS into a git repo for packages we want to change, generate a new SRPM after edits for those packages, and submit those modified SRPMS into the work queues. It also allows for the submission of non-modified SRPMS directly without the need to import them into git. It automates several things that we have done in the past by hand (automatically knowing which packages are not built by CentOS (for example the RHN packages that deal with upstream subscriptions) and automatically copies multilib 32bit packages into the 64 bit tree. The system also reliably produces the Yum-Presto DeltaRPMS and metadata for minimizing download times for updates.

    We do need to announce that CentOS-4 will be reaching the End Of Life at the end of February 2012. That means that there will be no more CentOS-4 updates after March 1st, 2012. If you are still using CentOS-4, you need to upgrade to CentOS-5 or CentOS-6 or switch to Red Hat's paid Extended Update Support for EL4 to continue to get updates. Please see the CentOS-4 EOL announcement for more details.

    So, news on the CentOS front for 2012 is very promising and we are looking forward to great things in the new year.


  • Stop Disabling SELinux!

    I see a lot of people coming by #centos and similar channels asking for help when they’re experiencing a problem with their Linux system. It amazes me how many people describe their problem, and then say something along the lines of, “and I disabled SELinux...”. Most of the time SELinux has nothing to do with the problem, and if SELinux is the cause of the problem, why would you throw out the extra security by disabling it completely rather than configuring it to work with your application? This may have made sense in the Fedora 3 days when selinux settings and tools weren’t quite as fleshed out, but the tools and the default SELinux policy have come a long way since then, and it’s very worthwhile to spend a little time to understand how to configure SELinux instead of reflexively disabling it. In this post, I’m going to describe some useful tools for SELinux and walk through how to configure SELinux to work when setting up a Drupal web site using a local memcached server and a remote MySQL database server -- a pretty common setup for sites which receive a fair amount of traffic.

    This is by no means a comprehensive guide to SELinux; there are many of those already!
    http://wiki.centos.org/HowTos/SELinux
    http://fedoraproject.org/wiki/SELinux/Understanding
    http://fedoraproject.org/wiki/SELinux/Troubleshooting

    Too Long; Didn’t Read Version

    If you’re in a hurry to figure out how to configure SELinux for this particular type of setup, on CentOS 6, you should be able to use the following two commands to get things working with SELinux:
    # setsebool -P httpd_can_network_connect_db 1
    # setsebool -P httpd_can_network_memcache 1

    Note that if you have files existing somewhere on your server and you move them to the webroot rather than untar them there directly, you may end up with SELinux file contexts set incorrectly on them which will likely deny access to apache to read those files. If you are having a related problem, you’ll see something like this in your /var/log/audit/audit.log:
    type=AVC msg=audit(1324359816.779:66): avc: denied { getattr } for pid=3872 comm="httpd" path="/var/www/html/index.php" dev=dm-0 ino=549169 scontext=root:system_r:httpd_t:s0 tcontext=root:object_r:user_home_t:s0 tclass=file

    You can solve this by resetting the webroot to its default file context using the restorecon command:
    # restorecon -rv /var/www/html

    Server Overview

    I’m going to start with a CentOS 6 system configured with SELinux in targeted mode, which is the default configuration. I’m going to be using httpd, memcached, and PHP from the CentOS base repos, though the configuration wouldn’t change if you were to use the IUS PHP packages. MySQL will be running on a remote server which gives improved performance, but means a bit of additional SELinux configuration to allow httpd to talk to a remote MySQL server. I’ll be using Drupal 7 in this example, though this should apply to Drupal 6 as well without any changes.

    Initial Setup

    Here we will setup some prerequisites for the website. If you already have a website setup you can skip this section.

    We will be using tools such as audit2allow which is part of the policycoreutils-python package. I believe this is typically installed by default, but if you did a minimal install you may not have it.
    # yum install policycoreutils-python

    Install the needed apache httpd, php, and memcached packages:
    # yum install php php-pecl-apc php-mbstring php-mysql php-pecl-memcache php-gd php-xml httpd memcached

    Startup memcached. The CentOS 6 default configuration for memcached only listens on 127.0.0.1, this is great for our testing purposes. The default of 64M of RAM may not be enough for a production server, but for this test it will be plenty. We’ll just start up the service without changing any configuration values:
    # service memcached start

    Startup httpd. You may have already configured apache for your needs, if not, the default config should be enough for the site we’ll be testing.
    # service httpd start

    If you are using a firewall, then you need to allow at least port 80 through so that you can access the website -- I won’t get into that configuration here.

    Install Drupal. I’ll be using the latest Drupal 7 version (7.9 as of this writing). Direct link: http://ftp.drupal.org/files/projects/drupal-7.9.tar.gz
    Download the tarball, and expand it to the apache web root. I also use the --strip-components=1 argument to strip off the top level directory, otherwise it would expand into /var/www/html/drupal-7.9/
    # tar zxf drupal-7.9.tar.gz -C /var/www/html --strip-components=1

    Also, we need to get the Drupal site ready for install by creating a settings.php file writable by apache, and also create a default files directory which apache can write to.
    # cd /var/www/html/sites/default/
    # cp default.settings.php settings.php
    # chgrp apache settings.php && chmod 660 settings.php
    # install -d -m 775 -g apache files

    Setup a database and database user on your MySQL server for Drupal. This would be something like this:
    mysql> CREATE DATABASE drupal;
    mysql> GRANT ALL ON drupal.* TO drupal_rw@web-server-ip-here IDENTIFIED BY 'somepassword';

    Test this out by using the mysql command line tool on the web host.
    # mysql -u drupal_rw -p -h drupal

    That should connect you to the remote MySQL server. Be sure that is working before you proceed.

    Now for the Fun Stuff

    If you visit your new Drupal site at http://your-hostname-here, you’ll be presented with the Drupal installation page. Click ahead a few times, setup your DB info on the Database Configuration page -- you need to expand “Advanced Options” to get to the hostname field since it assumes localhost. When you click the button to proceed, you’ll probably get an unexpected error that it can’t connect to your database -- this is SELinux doing its best to protect you!

    Allowing httpd to Connect to a Remote Database

    So what just happened? We know the database was setup properly to allow access from the remote web host, but Drupal is complaining that it can’t connect. First, you can look in /var/log/audit/audit.log which is where SELinux will log access denials. If you grep for ‘httpd’ in the log, you’ll see something like the following:
    # grep httpd /var/log/audit/audit.log
    type=AVC msg=audit(1322708342.967:16804): avc: denied { name_connect } for pid=2724 comm="httpd" dest=3306 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=system_u:object_r:mysqld_port_t:s0 tclass=tcp_socket

    That is telling you, in SELinux giberish language, that the httpd process was denied access to connect to a remote MySQL port. For a better explanation of the denial and some potential fixes, we can use the ‘audit2why’ utility:
    # grep httpd /var/log/audit/audit.log | audit2why
    type=AVC msg=audit(1322708342.967:16804): avc: denied { name_connect } for pid=2724 comm="httpd" dest=3306 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=system_u:object_r:mysqld_port_t:s0 tclass=tcp_socket

    Was caused by:
    One of the following booleans was set incorrectly.
    Description:
    Allow HTTPD scripts and modules to connect to the network using TCP.

    Allow access by executing:
    # setsebool -P httpd_can_network_connect 1
    Description:
    Allow HTTPD scripts and modules to connect to databases over the network.

    Allow access by executing:
    # setsebool -P httpd_can_network_connect_db 1

    audit2why will analyze the denial message you give it and potentially explain ways to correct it if it is something you would like to allow. In this case, there are two built in SELinux boolean settings that could be enabled for this to work. One of them, httpd_can_network_connect, will allow httpd to connect to anything on the network. This might be useful in some cases, but is not very specific. The better option in this case is to enable httpd_can_network_connect_db which limits httpd generated network connections to only database traffic. Run the following command to enable that setting:
    # setsebool -P httpd_can_network_connect_db 1

    It will take a few seconds and not output anything. Once that completes, go back to the Drupal install page, verify the database connection info, and click on the button to continue. Now it should connect to the database successfully and proceed through the installation. Once it finishes, you can disable apache write access to the settings.php file:
    # chmod 640 /var/www/html/sites/default/settings.php

    Then fill out the rest of the information to complete the installation.

    Allowing httpd to connect to a memcached server

    Now we want to setup Drupal to use memcached instead of storing cache information in MySQL. You’ll need to download and install the Drupal memcache module available here: http://drupal.org/project/memcache
    Install that into your Drupal installation, and add the appropriate entries into settings.php. For this site, I did that with the following:
    # mkdir /var/www/html/sites/default/modules
    # tar zxf memcache-7.x-1.0-rc2.tar.gz -C /var/www/html/sites/default/modules

    Then edit settings.php and add the following two lines:
    $conf['cache_backends'][] = 'sites/default/modules/memcache/memcache.inc';
    $conf['cache_default_class'] = 'MemCacheDrupal';

    Now if you reload your site in your web browser, you’ll likely see a bunch of memcache errors -- just what you wanted! I bet it’s SELinux at it again! Check out /var/log/audit/audit.log again and you’ll see something like:
    type=AVC msg=audit(1322710172.987:16882): avc: denied { name_connect } for pid=2721 comm="httpd" dest=11211 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=system_u:object_r:memcache_port_t:s0 tclass=tcp_socket

    That’s very similar to the last message, but this one is for a memcache port. What does audit2why have to say?
    # grep -m 1 memcache /var/log/audit/audit.log | audit2why
    type=AVC msg=audit(1322710172.796:16830): avc: denied { name_connect } for pid=2721 comm="httpd" dest=11211 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=system_u:object_r:memcache_port_t:s0 tclass=tcp_socket

    Was caused by:
    One of the following booleans was set incorrectly.
    Description:
    Allow httpd to act as a relay

    Allow access by executing:
    # setsebool -P httpd_can_network_relay 1
    Description:
    Allow httpd to connect to memcache server

    Allow access by executing:
    # setsebool -P httpd_can_network_memcache 1
    Description:
    Allow HTTPD scripts and modules to connect to the network using TCP.

    Allow access by executing:
    # setsebool -P httpd_can_network_connect 1

    Again, audit2why gives us a number of options to fix this. The best bet is to go with the smallest and most presice change for our needs. In this case there’s another perfect fit: httpd_can_network_memcache. Enable that boolean with the following command:
    # setsebool -P httpd_can_network_memcache 1

    Success! Now httpd can talk to memcache. Reload your site a couple of times and you should no longer see any memcache errors. You can be sure that Drupal is caching in memcache by connecting to the memcache CLI (telnet localhost 11211) and typing ‘stats’. You should see some number greater than 0 for ‘get_hits’ and for ‘bytes’.

    What are all these booleans anyway?

    Now we’ve used a couple SELinux booleans to allow httpd to connect to memcached and MySQL. You can see a full list of booleans which you can control by using the command ‘getsebool -a’. They are basically a preset way for you to allow/deny certain pre-defined access controls.

    Restoring default file contexts

    As I mentioned briefly in the ‘TL;DR’ section, another common problem people experience is with file contexts. If you follow my instructions exactly, you won’t have this problem because we untar the Drupal files directly into the webroot, so they will inherit the default file context for /var/www/html. If, however, you were to untar the files in your home directory, and then use ‘mv’ or ‘cp’ to place them in /var/www/html, they will maintain the user_home_t context which apache won’t be able to read by default. If this is happening to you, you will see the file denials logged in /var/log/audit/audit.log -- something like this:
    type=AVC msg=audit(1324359816.779:66): avc: denied { getattr } for pid=3872 comm="httpd" path="/var/www/html/index.php" dev=dm-0 ino=549169 scontext=root:system_r:httpd_t:s0 tcontext=root:object_r:user_home_t:s0 tclass=file

    The solution in this case is to use restorecon to reset the file contexts back to normal:
    # restorecon -rv /var/www/html

    Update: It was noted that I should also mention another tool for debugging audit messages, 'sealert'. This is provided in the setroubleshoot-server package and will also read in the audit log, similar to what I described with audit2why.
    # sealert -a /var/log/audit/audit.log

    Tags:



  • CentOS Continuous Release

    The CentOS Continuous Release repository (“CR”) was first introduced for CentOS 5.6, and currently exists for both CentOS 5 and CentOS 6. The CR repo is intended to provide package updates which have been released for the next point release upstream (from RHEL) which has not yet been officially released by CentOS yet due to delays around building, testing, and seeding mirrors for a new point release. For example, this means that once RedHat releases RHEL 5.8, CentOS will include package updates from 5.8 base and updates in CentOS 5.7 CR repo until the time that CentOS is able to complete the release of CentOS 5.8. For admins, this means less time without important security updates and the ability to be on the latest packages released in the latest RHEL point release.

    Details on the CR Repo

    What’s included in CR and how might it affect your current CentOS installs? At this point, the CR repo is used only for package updates which are part of the next upstream point release. For example, for CentOS 5.7, once Red Hat releases RHEL 5.8, the CR repo will contain updates from upstream base and updates repos. When a new update for RHEL 5.8 is released, it will be built in the CentOS build system, go through a relatively minimal amount of QA by the CentOS QA team, and then will be pushed to the CentOS 5.7 CR repo. This process will continue until the time that CentOS releases its own 5.8 release. Once CentOS releases 5.8, the CR repo will be cleared out until the time that RedHat releases the next (5.9) point release.

    The CR repo is not enabled by default, so it is up to a system administrator to enable it if desired. That means, by default, you won’t see packages added to the CR repo. Installing the repo is very easy as it’s now part of the CentOS extras repository which is enabled by default. To enable CR, you simply have to:

    yum install centos-release-cr
    

    If you don’t have CentOS Extras enabled, you can browse into the extras/ directory for the release of CentOS you’re currently running and download and install the centos-release-cr package by hand, or manually create a centos-cr.repo in /etc/yum.repos.d/

    In my opinion, unless you have an internal process for testing/pushing updates, you should absolutely be using the CR repo. Even if you do have your own local processes for updates, I would consider the CR repo to be part of CentOS updates for all intents and purposes, and pull your updates from there for testing/release. The packages in the CR repo can fix known security issues which without the CR repo you won’t have access to until the next CentOS point release -- and that can sometimes take longer than we’d like!

    A New Proposal: Include CR by Default

    In a recent post to the CentOS Developers list, Karanbir Singh proposed moving the CR repo into the main release for 6.x. What this would mean is for CentOS 6.x and onward, we would see the base OS and ISO directories be updated for each point release, but in general, updates would be pushed to a central 6/ directory, basically incorporating CR into what is currently considered updates/.

    This proposal is different from the current CR setup in that it incorporates CR into the release by default, and puts less reliance on the old point release model. This will help ensure that people are always running the latest security updates as well as take a bit of pressure off of CentOS developers and QA team when trying to build, test, and release the next point release. If the package updates are already released and in use, point releases become less important (though still useful for new installs).

    Incorporating CR more into the main release doesn’t mean that point releases will go away completely. They will still include updated base packages and ISO images, typically with installer bug fixes and/or new and updated drivers. In general, I see this as a good move: it means more people will be getting security updates by default instead of waiting during the time lapse between upstream RHEL releases and the time it takes for CentOS to rebuild, test, and release that point release. Having those packages available by default is great, especially for those admins who don’t pay close attention and wouldn’t otherwise enable the CR repo. It should be noted that at this point, the incorporation of CR into the main release is only being discussed for CentOS 6.x onward and won’t change anything in the 5.x releases where people will still need to manually opt-in to the CR packages.

    References:
    http://wiki.centos.org/AdditionalResources/Repositories/CR
    http://lists.centos.org/mailman/listinfo/centos-cr-announce
    http://lists.centos.org/pipermail/centos-devel/2011-November/008268.html

    Tags:



  • Corporate Security, Round Two
    I do not claim to be a security expert by any stretch of the imagination. The extent of my malicious network behavior ends at clicking 'start' on a nessus scan. And yet despite this, I find that I am constantly astounded at the inability of corporations to learn from each other when it comes to network security. Sony made security considerations front page news for over a month. Websites were

  • Monitoring DRBD resources with Zabbix on CentOS

    We use DRBD at work on several CentOS 5.x nodes to replicate data between our two computer rooms (in different buildings but linked with Gigabit fiber). It's true that you can know if something wrong happens at the DRBD level if you have configured the correct 'handlers' and the appropriate notifications scripts (Have a look for example at the Split Brain notification script). Those scripts are 'cool' but what if you could 'plumb' the DRBD status in your actual monitoring solution ? We use Zabbix at $work and I was asked to centralize events from differents sources and Zabbix doesn't support directly monitoring DRBD devices. But one of the cool thing with Zabbix is that it's like a Lego system : you can extend what it does if you know what to query and how to do it. If you want to monitor DRBD devices, the best that Zabbix can do (on the agent side, when using the zabbix agent running as a simple zabbix user with /sbin/nologin as shell) is to query and parse /proc/drbd . So here we go : we need to modify the Zabbix agent to use Flexible User Parameters, like this (in /etc/zabbix/zabbix_agentd.conf) :

    UserParameter=drbd.cstate[*],cat /proc/drbd |grep $1:|tr [:blank:] \\n|grep cs|cut -f 2 -d ':'|grep Connected |wc -l
    UserParameter=drbd.dstate[*],cat /proc/drbd |grep $1:|tr [:blank:] \\n|grep ds|cut -f 2 -d ':'|cut -f 1 -d '/'|grep UpToDate|wc -l

    We just need to inform the Zabbix server of the actual Connection State (cs) and Disk State (ds) . For that we just need to create Application/Items and Triggers .. but what if we could just create a Zabbix Template so that we can just link that template to a DRBD host ? I attach to this post the DRBD Zabbix template (xml file that you can import in your zabbix setup) and you can just link it to your drbd hosts. Here is the link . That XML file contains both two Items (cstate and dstate) and the associated triggers. Of course you can extend it, especially if you use multiple resources , drbd disks. Because we used the Flexible parameters, you can for example in the Zabbix item, create a new one (based on the template) and monitor the /dev/drbd1 device just by using the drbd.dstate[1] key in that zabbix item.

    Happy Monitoring and DRBD'ing ...



Follow us on Twitter

Visitor Data

Your IP
38.107.179.229
United States United States :
Browser
Unknown Browser Unknown Browser
Operating System
Unknown Operating System Unknown Operating System

Share this article:

Add to: Mr. Wong Add to: Webnews Add to: Icio Add to: Oneview Add to: Kledy.de Social Bookmarking Add to:  FAV!T Social Bookmarking Add to: Favoriten.de Add to: Seekxl Add to: Social Bookmark Portal Add to: BoniTrust Add to: Power-Oldie Add to: Bookmarks.cc Add to: Newskick Add to: Newsider Add to: Linksilo Add to: Readster Add to: Yigg Add to: Linkarena Add to: Digg Add to: Del.icoi.us Add to: Reddit Add to: Jumptags Add to: Upchuckr Add to: Simpy Add to: StumbleUpon Add to: Slashdot Add to: Netscape Add to: Furl Add to: Yahoo Add to: Blogmarks Add to: Diigo Add to: Technorati Add to: Newsvine Add to: Blinkbits Add to: Ma.Gnolia Add to: Smarking Add to: Netvouz Add to: Folkd Add to: Spurl Add to: Google Add to: Blinklist Information