Thanks for coming to visit onecool1 and see you around!

In business, to turn away just under 20% of your potential revenue to comply with your own principles must be a hard call to make. But Google is global, and perhaps 4.8 billion people in the rest of the world is a sufficient number to target with AdWords campaigns…

But what is really happening here? It’s difficult to believe that Google would invest so much time and effort, installing services in 2006, and then expect that within 4 years Beijing would accede to Google’s “wisdom” and suddenly allow freedom of speech. Within 4 years? After thousands of years of communist, dynastic and, occasionally, even tyrannical rule? No, somehow this seems unlikely.

It’s a surprising move by Google; one that could incite anything from a murmur of disquiet amongst the ranks of young Chinese teens, avidly seeking knowledge and understanding, to full-blown protests, perhaps even riots. It’s something of a political move, too: reading between the lines, it would appear that Google suspects Beijing of orchestrating the cyber-attacks on it and the twenty or so other organisations, as mentioned in their blog. By saying “play fair or don’t play at all”, Google may be vocalising the sentiments of the underclasses, still struggling to be heard from within the provinces.

Something that has not been mentioned (to my knowledge) so far in the press is the opportunity to expose Hong Kong. Under Chinese rule, but with special provisions (such as more liberal allowances on internet services), Hong Kong would present a potential new base for Google’s Chinese operation. But perhaps that’s a step too far?

The question remains whether it’s a viable exercise, and for viability, read “bottom-line”. Implementing the required censorship and publishing restrictions as required by the Chinese government will likely have been more technical trouble than they’re worth for Google, who elsewhere in the world have hands-down probably the most advanced information and revenue infrastructure to be found.

But information and revenue go hand in hand in Google’s business model. The less information, the less dynamism on-site, then the less interest there will likely be and the less uptake, over time. Google works in the west because there are virtually no limits, within the law, on trading ideas and services. In the far east, Google may have just observed a synergy that works to the detriment of its model. It may also be outgunned by larger powers at work; Beijing’s insurance.

We shall see if Google’s gambit, encouraging closer but more open ties with Beijing, will pay off.

The cause of the problem

I use SugarCRM for customer and project management data – and very good it is too! (Gratuitous plug – I can help your company install and use this fine software ). Except that recently, when listing my Accounts within Sugar, I would not see all of the account context. Only the account data itself would be displayed and none of the subpanels/links. The query to retrieve more data was failing, with this error message displayed in the browser window:

mysqld: Can't create/write to file '/tmp/#08y2jw' (Errcode: 13)

In my system log (/var/log/messages), I also got multiple SELinux errors like this:

Oct 13 09:07:50 server setroubleshoot: SELinux is preventing mysqld (mysqld_t) "read" to ./tmp (public_content_rw_t). For complete SELinux messages. run sealert -l 1762c478-f3a2-4eeb-be09-bd3dc037d945

Clearly, the reason for “Errcode: 13″ was due to SELinux.

Incidentally. if you have seen a similar error on your web site, but with (Errcode: 28) instead, this is likely due to shortage of disk space. A great way of determining operating system errors like this, is to use ‘PError’, thus:

# perror 28
OS error code 28: No space left on device

# perror 13
OS error code 13: Permission denied

So there we are – two distinct and different issues.

With SELinux, resolving the permission issue can be difficult. By issuing # sealert -l 1762c478-f3a2-4eeb-be09-bd3dc037d945, as suggested above, I got the following output (trimmed and highlighted for clarity):

Summary:

SELinux is preventing mysqld (mysqld_t) “search” to ./tmp (public_content_rw_t).

Allowing Access:

Sometimes labeling problems can cause SELinux denials. You could try to restore
the default system file context for ./tmp,

restorecon -v ‘./tmp’

Additional Information:

Source Context root:system_r:mysqld_t
Target Context system_u:object_r:public_content_rw_t

First things first: issuing # restorecon -v './tmp' didn’t fix it for me. I was also surprised to see that the path to /tmp was relative to the current working directory, so I tried a slightly modified # restorecon -v '/tmp', but to no avail. After restarting mysqld, the problem persisted: MySQL was simply being refused access to /tmp. Somewhere, a policy is disallowing this.

It’s a mistake to assume the the source context and target context should be the same; they don’t have to be, as it’s entirely policy-driven.  I made bold those aspects (the file Type) above to highlight this incorrect assumption (that I previously held).

Find and fix a policy?

Although finding the troublesome policy and analysing it is a Good Thing, it’s also time-consuming and requires significant knowledge of SELinux, chiefly to avoid creating security holes. A better way, I found, was simply to relocate where mysqld tries to store temporary data.

Thanks to Surachart Opun’s blog, I learned that you can specify a new location for temporary files. In /etc/my.cnf, add or edit the following:

[mysqld]
tmpdir=/tmp # <--- change this to a location that SELinux allows mysqld to write data.
# e.g.
tmpdir=/var/lib/mysql/tmp


Now do the legwork to set up the directory properly:

First, create directory with appropriate permissions
# cd /var/lib/mysql
# mkdir tmp
# chown mysql:mysql tmp
# chmod 1750 tmp

Now set the SELinux context up:
# chcon --reference /var/lib/mysql tmp

and make the SELinuiux context permanent:
# semanage fcontext -a -t mysql_db_t "/var/lib/mysql/tmp(/.*)?"

Finally, restart mysql:

SugarCRM - Open Source CRM system

# service mysqld restart

Closing thoughts: optimisation
The methods above fixed the particular problem I was having. They didn’t, however, actually pinpoint the cause. This is one of the good things about Linux and SELinux in particular: you are forced to rethink what the system is doing and work out a solution that sits within the predefined security context – or learn how to write SELinux policies. Personally, I prefer the former

There is an additional benefit to the solution above – namely, optimisation. Because we have specified the security context with semanage, we are free to mount an external file system and use that instead for MySQL’s temporary files. In other words, we can maintain the security but increase the performance.  One such filesystem could be tmpfs. tmpfs is actually a RAM Disk, uses a fixed amount of RAM to provide file storage. It is much quicker than an on-disk filesystem and thus perfectly optimised for storing temporary, caching data. There are many resources about tmpfs on the web. A good introduction to tmpfs can be at Planet Admon.

One of them is the plasma panel widgets – they can simply get messed up upon a “bad” log-out.

To fix your plasma taskbar/widgets getting messed up on KDE 4.2, and restore the default settings, follow these steps:

cd /usr/share/kde4/ kde-settings/kde-profile/default/share/configls
ls ~/.kde/share/config/
cat ~/.kde/share/config/plasma-appletsrc
mv ~/.kde/share/config/plasma-appletsrc ~/.kde/share/config/plasma-appletsrc.bak
cp plasma-appletsrc ~/.kde/share/config/

Below are the source repo definitions I use.  Simply create a file called “Centos-Source.repo” (# chmod 644) in /etc/yum.repos.d/ and enable repositories as required (using enabled=1).  Please note that this example is for CentOS version 5 and may differ from any official versions out there.  I offer no warranty… it just works for me. 

[base-SRPMS]
name=CentOS-$releasever – Base SRPMS
baseurl=http://mirror.centos.org/centos/$releasever/os/SRPMS/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5
priority=1
enabled=1

#released updates
[update-SRPMS]
name=CentOS-$releasever – Updates SRPMS
baseurl=http://mirror.centos.org/centos/$releasever/updates/SRPMS/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5
priority=1
enabled=1

#packages used/produced in the build but not released
[addons-SRPMS]
name=CentOS-$releasever – Addons SRPMS
baseurl=http://mirror.centos.org/centos/$releasever/addons/SRPMS/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5
priority=1
enabled=0

#additional packages that may be useful
[extras-SRPMS]
name=CentOS-$releasever – Extras SRPMS
baseurl=http://mirror.centos.org/centos/$releasever/extras/SRPMS/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5
priority=1
enabled=0

#additional packages that extend functionality of existing packages
[centosplus-SRPMS]
name=CentOS-$releasever – Plus SRPMS
baseurl=http://mirror.centos.org/centos/$releasever/centosplus/SRPMS/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5
priority=1

#contrib – packages by Centos Users
[contrib-SRPMS]
name=CentOS-$releasever – Contrib SRPMS
baseurl=http://mirror.centos.org/centos/$releasever/contrib/SRPMS/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5
priority=1

I recently installed Fedora 11 on a test machine, to see how the development desktop build of my favourite Linux distribution is doing – and it’s very nice indeed.

I tried to install Zend Studio 5.5 and soon came across problems, which I found out related to not having 32-bit versions of Xorg and glibc installed. To remedy this, ensure you follow these steps:
su -c 'yum groupinstall Java'
su -c 'yum install glibc.i686 libXext.i586'
Once installed, I was able to fire up the Zend Development Environment:
/usr/local/Zend/ZendStudio-5.5.1/bin/ZDE
.. and it was running on the native (OpenJDK) 64-bit JAVA runtime! How’s that for progress!

What next? If, like me (on one occassion) you try to boot up the machine and get no further than the recovery console, you’d feel a bit aggravated. But there is an alternative – do a disk check. You may have read my verbose coverage for How to do a disk check in Linux before. This takes it one step further – how to check your logical volume when it’s encrypted and formatted using the latest ext4 filesystem.

Instead of the method used before, this time I booted from a Live CD. You can find one to download at the Fedora project. Ensure that this CD matches the release of the version you are trying to recover. In this case, that’s Fedora 10.

Once you have booted the offending machine up with the Live CD, open up a terminal by pointing to Applications > System Tools > Terminal. Once in the terminal window, just type:

# su

…to become the root user. This is essential to using all the disk tools.

You may be tempted to check for volume groups first:

# vgscan

.. but this would return nothing.

What’s happening here is that the Volume Group, located on that partition, is itself encrypted. Once unlocked, you can then gain access to both of the Logical Volumes – the swap volume and the root (/) volume.

To unlock the encrypted Volume Group, first you need to establish which partition it resides on:

# fdisk /dev/sda

The number of cylinders for this disk is set to 12161.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)

Hit p <enter> to print the partitions on your primary disk:

Command (m for help): p

Disk /dev/sda: 100.0 GB, 100030242816 bytes
255 heads, 63 sectors/track, 12161 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0xb07eb07e

Device     Boot  Start End    Blocks   Id System
/dev/sda1    *       1 5377   43190721  7 HPFS/NTFS
/dev/sda2         5378 5402    200812+ 83 Linux
/dev/sda3         5403 12161 54291667+ 8e Linux LVM

So the partiton of type “Linux LVM” (Logical Volume Managed) is the baby we’re after.

To unlock the encrypted Volume Group, use the following:

# cryptsetup luksOpen /dev/sda3 mydisk

This sets up the encryption/decryption kernel subroutines to allow access to device /dev/sda3, mapped to a device node called “mydisk” in /dev/mapper/ . We’ll not actually need to use this device node, but it could be handy to know if you needed to perform further diagnostics.

You will be prompted to enter the encryption key which is stored in one of eight “slots” on the disk. If the key you enter matches a key in any slot, your disk will become unlocked. Assuming that it is, you can then scan once again for Volume Groups:

# vgscan
Reading all physical volumes. This may take a while...
Found volume group "VolGroup00" using metadata type lvm2

Now we’re getting somewhere. Let’s activate the VG and display the LVs (Logical Volumes) it contains:

# vgchange -a y
2 logical volume(s) in volume group "VolGroup00" now active
# lvdisplay
--- Logical volume ---
LV Name /dev/VolGroup00/LogVol00
VG Name VolGroup00
LV UUID RE7t2h-nIy9-dWZ9-xt26-Fgq4-gFd8-34E3f2
LV Write Access read/write
LV Status available
# open 0
LV Size 47.81 GB
Current LE 1530
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:3

--- Logical volume ---
LV Name /dev/VolGroup00/LogVol01
VG Name VolGroup00
LV UUID B7XJzD-9sS0-3iHx-AWBE-W9qN-TvRb-vCdYZF
LV Write Access read/write
LV Status available
# open 0
LV Size 3.91 GB
Current LE 125
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:4

We can deduce from the sizes of these two volumes that the first of the two is the root (/) volume, and the second is the swap volume.

As the purpose is to FIX the filesystem on it, which may have become corrupt through the hard-reset performed earlier, we do not want to mount this volume. Instead, as we now have a device node for this activated volume, at /dev/VolGroup00/LogVol00, we can simply perform a disk check straight on it.

To check which extn file system checking tools are on the system, you can tab-complete at the command line:

# fsck. (hit tab)
fsck.cramfs fsck.ext3 fsck.ext4dev fsck.vfat
fsck.ext2 fsck.ext4 fsck.msdos

As this was formatted an ext4 volume, that’s what we use:

# fsck.ext4 /dev/VolGroup00/LogVol00
esfsck 1.41.3 (12-Oct-2008)
/dev/VolGroup00/LogVol00: recovering journal
Clearing orphaned inode 730 (uid=0, gid=500, mode=0100600, size 2263160)
Clearing orphaned inode 187182 (uid=500, gid=500, mode=0100600, size 4096)
... and so on until ...
/dev/VolGroup00/LogVol00: clean, 190926/3137536 files, 2016683/12533760 blocks

Now there are two more steps to perform: de-activate the volume group, and lock the encryption of the drive.

# vgchange -a n
0 logical volume(s) in volume group "VolGroup00" now active
# cryptsetup luksClose mydisk

The second command returns nothing, which means it did not error (the disk is now encrypted and not writable-to without unlocking again).

I hope that helps someone with a sense for adventure but not enough time on their hands for when things go somewhat awry!

So how does one go about it on Linux?  Well… as you may have guessed, it’s not quite so straightforward.  Linux, by default, does actually have an intelligent disk-checking system already in place. By all accounts, you generally needn’t worry.  But if you have a reason to believe your disk may be slowly dying, and nothing is reporting in the SMART status of your drive, perhaps it’s worth checking the file system instead.

That’s where File System Check comes in (duh!).  Like all Linux tools, it’s painfully abbreviated to simply “fsck”.  Terse, to say the least.  Now the warning:

DO NOT.  I REPEAT, DO NOT EVER EVER EVER RUN THIS COMMAND WHILE YOUR DRIVE IS MOUNTED (I.E. IN USE).  I TAKE NO RESPONSIBILITY FOR ANY LOSS OF DATA THAT YOU MAY CAUSE BY FOLLOWING THESE INSTRUCTIONS.

To unmount your root (/) volume, follow these easy steps:

1. Boot from a Live CD. Your root volume will not be mounted by default.
2. Open a terminal and type:# dmesg | grep sda

   If you see output relating to your “SCSI” device, then this will identify that your hard disk, in all likelihood, contains your root partition. For example, amongst other output, I see this:

   sd 2:0:0:0: [sda] Assuming drive cache: write through
sda: sda1 sda2
sd 2:0:0:0: [sda] Attached SCSI disk


3. In the example above, we see that SCSI disk 2 (2:0:0:0:) the Linux kernel registers it as the first logical drive (“sda”) in the system.  We can also see it has only 2 partitions, sda1 and sda2.  If this is the only physical drive in the machine, we should strongly suspect that it uses one partition as /boot (formatted with ext3) and the other as a Logical Volume containing both root (/) and swap. Furthermore, it’s foregone conculsion that the smallest partition will be /boot and the larger one will contain our swap and / partitions, so let’s proceed with accessing them.
4. So, how do we access a “Logical Volume” within an equally mystical “Volume Group”?  Luckily, Linux LVM comes with a plethora of useful tools to make the job easy.
   
# /sbin/vgscan
Reading all physical volumes. This may take a while...
Found volume group "VolGroup00" using metadata type lvm2

   Great. We have identified the volume group.  But before we can identify the logical volumes it contains, we need access it.

   # /sbin/vgchange -a y
2 logical volume(s) in volume group "VolGroup00" now active

   Here, the -a flag indicates that we want to change the “active” status of the volume group, and the y means “yes”.

   # /sbin/lvdisplay
--- Logical volume ---
LV Name                /dev/VolGroup00/LogVol00
VG Name                VolGroup00
LV UUID                DG2WxJ-sKa5-20mg-NtjW-CsPW-t99V-Egqlja
LV Write Access        read/write
LV Status              available
# open                 0
LV Size                7.25 GB
Current LE             232
Segments               1
Allocation             inherit
Read ahead sectors     auto
- currently set to     256
Block device           253:2

   --- Logical volume ---
LV Name                /dev/VolGroup00/LogVol01
VG Name                VolGroup00
LV UUID                HqKozT-16PQ-HUaT-Yyc7-lMCO-007m-Xcc2c8
LV Write Access        read/write
LV Status              available
# open                 1
LV Size                512.00 MB
Current LE             16
Segments               1
Allocation             inherit
Read ahead sectors     auto
- currently set to     256
Block device           253:3

   We can now see two partitions contained within the volume group. The first partition, although small by today’s standards, looks a lot larger than the second.  We can also see that each logical volume has a device node (/dev/VolGroup00/LogVol01, for example).

   As we want to perform the disk check without the parition being mounted, we do not issue any mount command here.  However, if you wanted to double-check that this is the partition to check, mount it and have a quick look around.  The following step is only offered to help in this case – skip this if you wish to perform a disk check.

   # mkdir /tmp/lv0

   For me, the first logical volume (the 7.5GB one) would be the one to test.

   # mount -t ext3 /dev/VolGroup00/LogVol00 /tmp/lv0
# cd /tmp/lv0
# ls
bin  boot  dev  etc  home  lib  lib64  lost+found  media  mnt  opt  proc  root  sbin  selinux  srv  sys  tmp  usr  var

   Ok, that looks like the root partition, so let’s get out of it and unmount it before running the file system check on it.

   # cd /
# umount /tmp/lv0

5. An alternative to the above steps, if you have already booted into your main system, is to investigate /etc/fstab to see which is your / volume.  All you do is open a terminal and issue: # cat /etc/fstab On my CentOS 5.2 system, I see this:/dev/VolGroup00/LogVol00 /                      ext3    defaults        1 1
LABEL=/boot1            /boot                   ext3    defaults        1 2
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
LABEL=SWAP-sdb1         swap                    swap    defaults        0 0

   So, /dev/VolGroup00/LogVol00 is my root volume.

So, now that that’s out of the way, what next?  Well, assuming you now know which is your root partition, the most sensible thing to do would be to boot from a Live CD of some distribution (Ubuntu, Fedora, etc) if you haven’t done so, and then perform the disk check from that.

Once in the LiveCD desktop, we’ll need to fire up a Terminal window.

If you know your filesystem type, e.g. if it’s Ext3, which is the default on the most common distributions, you can run a modified version of the fsck command specifically for that file system.  Here’s what I run for a thorough disk check:


# fsck.ext3 -C -D -f -P -v /dev/VolGroup00/LogVol00

Alternatively, if your partition structure is slightly older and only contains physical paritions (not Logical Volumes), it may just be a case of finding the partition directly – by checking /etc/fstab on the system when running. In that case, your command may look more like this (when / is unmounted!!):

# fsck.ext3 -C -D -f -P -v /dev/sda2

Here’s what the flags do:

-C  – forces a bad block scan.  Although bad blocks are remapped dynamically by the file system, if the file system or its journal are corrupt, this may not work correctly.

-D  – performs a directory check and optimisation.  Doesn’t hurt, and can speed up directory listings of a large number of files.

-f  – forces the check itself to actually run.  As mentioned previously, the file system maintains itself quite well, and if you don’t force the check, fsck may look at the last check interval and decide a check is not required.

-P  – perform all file system fixes automatically.  This is usually a safe flag, but if your file system is potentially very corrupt, this may not be advisable.  In this situation, contact an expert – or restore your back-up…

-v   – verbose output. See what’s going on.

/dev/VolGroup00/LogVol00 or /dev/sda2 – this is the partition I want to perform the disk check on.

This little guide doesn’t explain how to perform a check on an encrypted logical volume… That one’s coming.

As a Microsoft Windows Small Business Server 2003 user, I have the option of using either Microsoft Outlook for native Exchange mail server connectivity, or using another, open standard protocol such as IMAP.  So, in my finite wisdom, I decided… why not?!

Why not indeed.  The reason for using IMAP (Internet Message Access Protocol) is so that I could use….. wait for it… a non-Microsoft email client with my Microsoft server.  The very notion.  Well, actually it’s not too bad.  You enable the IMAP service on the server, set up the mail account in Thunderbird, and hey presto – log in!

Unfortunately, my client (Mozilla Thunderbird) then seemed to have done certain things which – only now – take my slightly by surprise.

An Exchange mailbox, as standard, contains some basic top-level folders, such as Calendar, Contacts, Deleted Items, Drafts, Inbox, Outbox, Sent Items and Tasks (this is probably not an exhaustive list).  In contrast Thunderbird, by default, contains Inbox, Drafts, Sent, Deleted, Junk and Trash.  So, what’s in a name?

Well, after using Thunderbird/Exchange via IMAP (and not actually noticing this at the time of doing so), a couple of the Exchange folders had disappeared.  I only noticed this later when using Outlook again, and couldn’t locate my Sent Items or Deleted Items folders.  I then found them lurking within my Trash folder.  Ok, so this has got very messy.

It sadly got worse.  Now that these “Special Folders” in Microsoft parlance have been moved, they could not be moved back in Outlook.  When trying to drag “Deleted Items” to my top-level Mailbox, I would be told “Cannot move special items.  Special folders, including the Inbox, Contacts, Calendar, Notes, Tasks and Journal folders, cannot be moved.”  Oh, I see.   Although I tried various methods within Outlook to achieve the same thing, I failed miserably.

So what is the solution?  Ironically, going back to Thunderbird and simply dragging the folder from Trash into the top-level mail account/box did it.  It re-sync’ed over IMAP and everything gets copied correctly.  How ridiculous.  The solution is not to run scanpst.exe or scanost.exe, or to start up Outlook using the “Safe” switch, thus:

C:\Program Files\Microsoft Office\Office12\OUTLOOK.EXE” /safe

It’s not even starting up Outlook with “Reset Folders”:

C:\Program Files\Microsoft Office\Office12\OUTLOOK.EXE” /resetfolders

… or the combination of both.

This is one of those annoying, inexplicable problems that you somehow just get used to.  I hope this helps someone else out there who has suffered the same issue.