GenerationIP

"Anyone who considers arithmetical methods of producing random digits is, of course, in a state of sin"
  -- John von Neumann

Stipulated, but I am beset by closer devils, and I've tinkered with a mild solution I like. Let me tell you more

I've been badgered as some web sites have moved to Java-script evaluation routines for site passwords. Some require mixed case; others punctuation; no doubled letters; minimum length. Contrariwise some limit the character set to prohibit what others require

As such, for the last year or so I have been playing with a quick script on my CentOS 5 box, to generate a unique password per website, and keep a master index of the userid, email address used, and password used. This of course limits my ability to connect to those sites when away from that list. Open-ID roll-in seems to be coming however, and I have a rather clever device, backed by Verisign, and using an inexpensive OTP -- one time password -- hardware device as part of the authentication process

The little generator I wrote is based on simple shell tools -- md5sum, df, date, ps, cut, tr and so forth. It gathers a bit of entropy from a few sources froma few systems around the office, which should be non-correlated from a theoretical basis in the time frames at issue. It does some hashing to get good dispersion. Then it expands into 3 or 4 character vectors each 16 characters wide, using the hexidecimal digits that md5sum emits, as translated by tr; the first three are letters, upper, lower, and digits; the fourth character set are selected specials and punctuation excluding some shell meta's. Depending on a limitation by an option to -a, that vector may be limited to the alpha-numerics only, or also stir in the specials

That 'deck of characters' is handed off to a 'repeated cut of the deck' shuffler, and returned mixed once more just for good measure

I then add a 'bumper' of a letter or digit at each end [one site prohibited starting with a special], and a second character of 'bang' to prevent a mouse slip from dropping a password into the bash history in the case of a panel slip

The results are assembled, trimmed to an optionally specified length, and displayed, where I harvest them as mentioned above

Really, passwords need to die, die, die, but that is for another post

Completely un-memorizable of course, so really only suitable to a protected physical environment where one may write them down

Random number sin, of course, but the cyber-ninja can more readily put my thumb between pliers jaws, than predict the pseudo random source values I used, I'll readily spill the secret to access my LOL CATS site account. ... I still have to get around to building a few non-correlated hardware random number generators -- diode based, lava lamp based, dice tumbling machine for serious entertaining, I guess

His mother must be so proud to have raised such a pottymouth:


Date: Tue, 27 Jul 2010 15:01:39 +0000
From: BOBBY RAY MCALLISTER
To: centosweb@centos.org
Subject: www.centos.org - Contact the CentOS WebMaster Form

BOBBY RAY MCALLISTER submitted the following Information:
Email BLACKTHORNE4440@AOL.COM
URL AIN'T GOT ONE
ICQ NONE OF THEM, EITHER
Company AIN'T WORKING
Location U.S. OF FUCKING A.
Comments

NO MORE. NO FUCKING MORE OF YOUR BULLSHIT SOLICITATIONS FOR "THIRD PARTIES"
TO ME, MOTHERFUCKERS. THE NEXT ONE GETS COPIED TO THE FTC AS WELL AS FBI
FOR PROSECUTION UNDER ANTI-SPAM STATUTES. I HAVE ASKED YOU TO MAKE FUCKING
WELL SURE TO DELETE MY ADDRESS, AND I STILL GET BULLSHIT FROM YOUR STUPID
FUCKING ASSHIOLE MOTHERFUCKING CLIENTS.

Mozilla/4.0 (compatible; MSIE 7.0; AOL 9.0; Windows NT 5.1; Trident/4.0;
GTB6.5; .NET CLR 1.0.3705; .NET CLR 1.1.4322; Media Center PC 4.0; .NET CLR
2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)


The template CentOS placeholder web page, which is found when people place IP addresses from mail headers into a web browser address bar and hit an unconfigured system, makes it quite clear that we merely supply an operating system, of course. I thought we went through all of this years and years ago with the Tuttle, Oklahoma to and fro. Reading with comprehension must be a rare skill; civility, rarer still

The end headers indicate this person has 'hit the trifecta': Windows, AOL, and Media Center Sadly, the salesperson who sold him those fine products must have also sold him a teletype as the console to use as well, as it is in ALL CAPS

Amazing stuff. I see that I need to amend that form to show originating IP, and perhaps put it under a 'captcha' to ensure at least some ability to read before posting

I write about CentOS, annoyances, and nuisances all the time. 'Lest people consider me a grumpy old man, never pleased, I'll drop my guard a bit here and now, and perhaps just this once

I just posted this in Joe Landman's comments section to his blog, but it might get lost. It bears repeating

Russ Herrold says:
July 26, 2010 at 6:23 pm

Hi, Joe

I have not had the chance to write the blog post, but it will get written.

My Dell laptop died, and I sent it to the following folks for refurb, based on some strong recommendations in a TX LUG mailing list

It came back fine, but then died.

http://twitpic.com/20p796



I was a slacker and did not ship it back right away. Even after the warranty interval expired, however they took it thru their intake, ID’d a bad video card they had added during the refurb, and swapped it out. gratis

And then shipped it back to me FedEx ... again gratis

One guess who I’ll be using from now on, for out of warranty repairs of Dell kit

http://www.parts-people.com/
512-339-1990

Tell them I sent you ;) Please say 'thanks again' for me

– Russ herrold

The primary usage case is we describe is how to deploy a read-only FTP server with no end user accounts, to be used for distribution of content (here, to be a 'hotfix' archive for publicly accessible binary updates, accessible through yum). We need this to work around a temporarily broken update in CentOS space. We can also use it to add additioanl packages under and under the mediation of the rpm package database

We start with a hardened PMman instance. A secondary purpose of this post is to work from first principles through adding a proper local 'forked packages' archive for CentOS users to follow as a worked example. At all times, we will strive to follow proper sysadmin 'best practices' discpline under SElinux, wrappers and iptables

1. Install and enable vsftpd which is the package holding the stock ftp daemon -- yum can do this trivially

   yum install vsftpd

   Then enable the ftp server:

   /sbin/chkconfig vsftpd on

   and create a pilot file to look for in later testing:

   mkdir -p /var/ftp/pub/mirror
echo test > /var/ftp/pub/mirror/README.txt

2. Run updates, just 'because' and as a matter of good sysadmin

   yum update
yum clean all

3. Open wrappers to permit anonymous FTP connections. We edit /etc/hosts.allow and add:

   vsftpd: ALL@ALL

4. Amend the iptables rules to allow ftp. The file /etc/services reminds us that FTP normally lives at TCP port 21

   1. Add to /etc/sysconfig/iptables-config to include 'ip_conntrack_ftp' in the list of 'IPTABLES_MODULES='

      IPTABLES_MODULES="ip_conntrack_ftp "

   2. and then, in /etc/sysconfig/iptables we add a line to pass FTP content:

      -A RH-Firewall-1-INPUT -m state --state \
   NEW -m tcp -p tcp --dport 21 -j ACCEPT

      [Note: We use the backslash convention here, but iptables does not support this in its config files]

5. Run the unit through a reboot, both to 'set' the updates by stopping use of any libraries held open through that update, and also to ensure that it works as expected after a 'hands off reboot'

6. Test from a remote host that FTP works as expected

   [herrold@centos-5 ~]$ lftp 198.49.244.190
lftp 198.49.244.190:~> cd /pub/mirror
cd ok, cwd=/pub/mirror
lftp 198.49.244.190:/pub/mirror> ls
-rw-r--r-- 1 0 0 5 Jul 20 16:56 README.txt
lftp 198.49.244.190:/pub/mirror> cat README.txt
test
5 bytes transferred
lftp 198.49.244.190:/pub/mirror> exit
[herrold@centos-5 ~]$

   ... great

At this point, we have a working RO anonymous ftp server, and can populate it with content.

As some of you already know it, DRBD is now (since kernel 2.6.33) part of the mainline/upstream kernel. Some were expecting RHEL6 to come with that kernel (used for Fedora 13). The latest RHEL6beta2 still comes with 2.6.32, which doesn’t include DRBD support. Of course we still don’t know what the ‘frozen’ RHEL6 kernel will be but on the other hand, we know that Red Hat quite often ‘backports’ modules from newer kernel into the RHEL kernel. What about DRBD ? At the time of writing this blog post, it seems still undecided, but you can follow the DRBD RFE on Upstream Bugzilla to get a clue, or even comment on it if you have a bugzilla account to  make hear your voice. On the other hand, you can still be sure that even if DRBD isn’t part of EL6, CentOS will still ship it in the Extras repository, like for EL4/EL5 …

I had to build a fresh set of rpms for the area_cli tools, and decided it was also a good time to rebase some of my local personal build tools to rhel6beta2 since its there and I need to do some tests with it anyway.

Firstly, there are a couple of cool things with rhel6 beta2 - rpmdevtools is included by default, which helps gets things started and helps a bit in managing spec files etc.

What isnt so cool is that the newer rpm in el6beta creates packages which then cause issues with some of the older CentOS releases. eg. this areca_cli package built on el6beta2/x86_64 for target i686 caused this to happen with yum:

Downloading Packages:
areca_cli-1.83_091103-1.e 100% |=========================| 493 kB    00:00     
Running rpm_check_debug
ERROR with rpm_check_debug vs depsolve:
rpmlib(FileDigests) is needed by areca_cli
rpmlib(PayloadIsXz) is needed by areca_cli
Complete!
(1, [u'Please report this error in https://bugzilla.redhat.com/enter_bug.cgi?product=Red%20Hat%20Enterprise%20Linux%205&component=yum'])

Which causes the package to not install. This is on a CentOS-5.2 machine. And I need to maintain that at CentOS-5.2 due to various issues ( no, machine is not available on the internet, and only hosts a single app in production ). Trying the install on a CentOS-5.3 machine I get the exact same issue. Upgrading rpm to 4.4.2.3-18.el5 ( the version in 5.4 and 5.5 ) makes no difference.

On the other hand, builds run on CentOS-5 have no problems installing on EL6Beta. So for the time being it looks like all buildhosts and all build stuff will need to stay on EL5. Specially since that allows you to target CentOS-3 and CentOS-4 as well.

- KB

Note: yeah, I see that url pointing at bugzilla isnt idea. I'll look into plumbing in a bugs.centos.org reference instead.

grep vsftpd /var/log/audit/audit.log | \
   audit2allow -M vsftpd
semodule -i vsftpd.pp
vi vsftpd.te
checkmodule -M -m -o vsftpd.mod vsftpd.te
semodule_package -o vsftpd.pp -m vsftpd.mod
semodule -i vsftpd.pp

More information that is accurate is better than less. Clearly there are many paths to rule generation and maintenance. The takeaway remains: Use, and do not disable, SELinux

Thanks for the feedback

Rusty Coker mentioned in a recent blog post that he had not found a COLO facility or VM provider that enabled SELinux in its hosts by default. People regularly whine: It's too hard, and I don't need it and disable the SELinux protections. Foo

I call: Bull on the latter As to the former I sent a private email to Rusty, and offered to 'comp' him an instance to break

If anyone knows of a virtual hosting company that runs Xen or KVM virtual machines with SE Linux support then please let me know, I'll write a blog post comparing such companies if there are some.

umm -- I would be embarrased to be a hosting provider which did NOT enable SElinux

Please feel free to set up a 'comp' account at:
http://www.pmman.com/signup/
at the green arrow. Use the [please do not repeat this] 'Offer Code' of: ...

... I repeated the offer at his blog's comment site

And the question came up today in the #centos IRC channel

13:52 Andro1d> orc_orc: how can i recompile a pp from a te ?
13:53 Andro1d> checkmodule -M -m -o vsftpd.mod vsftpd.te gives a lot of errors :-/
13:53 orc_orc> ehh?
13:53 wolfy> Andro1d:
http://wiki.centos.org/HowTos/SELinux [CentOS wiki]
13:53 orc_orc> make a working dir -- say:
mkdir -p /etc/selinux/targeted/foo
and cd into it
13:54 orc_orc> Gather all the selinux noise:
audit2allow -i /var/log/audit/audit.log* -m local > local.te
13:54 Andro1d> hm, I think I'm missing some types in my .te file
13:54 orc_orc> Note the '*' in that prior line, which reads all log files present
13:54 Andro1d> mom...
13:54 orc_orc> Install the selinux-devel package for the needed Makefile
13:54 Andro1d> don't wanna make a "huge" selinux policy :)
13:54 orc_orc> Then run:
make -f /usr/share/selinux/devel/Makefile
13:55 orc_orc> and apply it:
semodule -i local.pp
13:55 orc_orc> Test again
13:55 Andro1d> yop, mompl
13:55 orc_orc> When happy, be sure to save a versioned copy, because SELinux audit file ageing will cause you to forget what was needed in that merge
13:55 orc_orc> For extra credit, amend:
/etc/audit/auditd.conf
to retain a sensible universe of back logs
13:56 orc_orc> '4' is wayyy too small

wolfy (a channel regular who offers reliable answers), pointed to the CentOS secondary source answer in the wiki; this post will also pass into our planet as yet another piece of documention and 'cheatsheet'. You saw a self-described RHCE (and he was proud of it coming into the channel today) doing that whimpering for his mommy as I read him the 'riot act'. I don't care in the least that this is new and 'hard' -- growing and learning new tools is part of the Unix culture, always has been, and always will be. That is why I try to make #centos a learning venue rather than a drive-by 'spoon-feeding' shop

How many times do we need to bang the SELinux drum to get your attention?

Yes, you lazy slogs of alleged sysadmins who simply disable SELinux, I am talking to YOU! yep - words are hard to memorize, but this is a basic 'lather, rinse and repeat' cycle which one can solve experimentally if not predictively from knowledge of what is happening. Run a tail -f /var/log/audit/audit.log if you must to see when the rule set needs to be rebuilt

But stop disabling SELinux and stop making excuses

It's been a long June. I noticed early on that an update in Debian testing had moved mke2fs from one package to another without getting all the library dependencies right. As such I spent June without the ability to lay down a filesystem on a new partition with the 'proper' tool. Part of my series on logfile reading includes a task to review the 'percent full' for each partition (and to relocate or clean out fat ones) to avoid running out of room in a self-inficted denial of services attack

I tried the obvious fallback to build a new filesystem: busybox but the version found in Debian Testing was lacking a needed build time switch. I filed the bug, and considered a local patch, or perhaps whether to rebuild of part of the chain needed to fork mkfs for a bit, but my need for space to reorganize a host's files was not that great nor urgent. Just pesky each day to see

I knew from reading the bug reports that the fix had been committed and 'ageing' in the Debian fashion to its move from an Unstable 'nightly' to a mildly tested (or at least not black-balled) state and promotion into Testing

nfs2:~# apt-get upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages have been kept back:
ksysguard libdevmapper1.02.1
The following packages will be upgraded:
bsdutils e2fslibs e2fsprogs iptables iso-codes libblkid1 libcomerr2
libenchant1c2a libffcall1 libmime-tools-perl libnetpbm10 libss2 libuuid1
lockfile-progs mount mutt netpbm shared-desktop-ontologies util-linux
19 upgraded, 0 newly installed, 0 to remove and 2 not upgraded.
Need to get 9,841kB of archives.
After this operation, 115kB disk space will be freed.
Do you want to continue [Y/n]? y
...
nfs2:~#

I've been running repository data update operations daily .. the Debian approach is more measured in its pace than we use with CentOS, and I think we may have something to learn there. It is a rare package update that cannot wait for a daily repo data update, push and mirror overnight in our space, and it would avoid much confusion to casual sysadmins

Those bolded packages in that clutch of upgrades looks promising ...

nfs2:~# mkfs /dev/sda12
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
237568 inodes, 949835 blocks
47491 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=973078528
29 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736

Writing inode tables: done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 28 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
nfs2:~# date
Thu Jun 24 10:13:17 EDT 2010
nfs2:~#

Lovely; I'm back in business

It looks like I'll be writing these for a while as I clean up logfile noise. The earlier pieces are here and here. I say 'noise' here because they are not false positives, but neither are they material, just more a nuisance

One the things every admin who reads log files sees are automated scanners looking for exploits in 'canned' packages that were installed but have not been updated, either because the admin for a given machine has neglected to run updates, because it is not a publicly known exploit, or because the upstream has not yet addressed the matter.

A pattern that has emerged with our PMman with a data center with large contiguous swaths of IP space (and hosts scattered in assignment in that relatively compact range, said hosts reporting to me centrally) is as follows. The hostile exploit scanners are not even trying to be subtle any more -- they simply march sequentially through IP ranges, and inventory if a given weakness is present on every host to which they connect

Today, I focus on one sample report stanza:

--------------------- httpd Begin ------------------------

Requests with error response codes
400 Bad Request
HTTP/1.1: 1 Time(s)
403 Forbidden
/index.html: 1 Time(s)
404 Not Found
/cms/e107_files/e107.css: 1 Time(s)
/db/e107_files/e107.css: 1 Time(s)
/e107/e107_files/e107.css: 1 Time(s)
/e107_files/e107.css: 1 Time(s)
/forum/e107_files/e107.css: 1 Time(s)
/index.php: 1 Time(s)
/manager/html: 1 Time(s)
/portal/e107_files/e107.css: 1 Time(s)
/site/e107_files/e107.css: 1 Time(s)
/web/e107_files/e107.css: 1 Time(s)

---------------------- httpd End -------------------------

and apache can handle this trivially:

#
# file: noexploit.conf
#
# send scanners off to see the wizard
#
Redirect permanent /cms http://127.0.0.1/
Redirect permanent /db http://127.0.0.1/
Redirect permanent /e107 http://127.0.0.1/
Redirect permanent /forum http://127.0.0.1/
Redirect permanent /manager http://127.0.0.1/
Redirect permanent /mysql http://127.0.0.1/
Redirect permanent /phpmyadmin http://127.0.0.1/
Redirect permanent /phpMyAdmin http://127.0.0.1/
Redirect permanent /portal http://127.0.0.1/
Redirect permanent /site http://127.0.0.1/
Redirect permanent /user http://127.0.0.1/
Redirect permanent /users http://127.0.0.1/
Redirect permanent /web http://127.0.0.1/
#

The obvious next step is to package deployment hardenings, and add them to a local RPM repository so that simply running updates, as with yum will get the current best approaches on hardening, en masse, on all the servers