LJDT: RPM Package Verification and Repair
As everybody who uses a computer in a multi-user environment eventually finds out permissions on files and directories sometimes get in your way. This is typically a good thing since they also get in the way of things like viruses and malware and other users on the system trying to get your stuff or break your computer. Windows users, well, you haven’t perhaps dealt with this much in life since windows is a bit of a red light district (and I mean that in a bad way) as you probably also already know.
So when permissions become torn up on your precious application files because of a bad set of instructions, a typo, or some other glitch in software (it happens, even with Linux), what is a user to do? The first thing that comes to mind may be to have a backup that you can restore, but that implies several things which I do not want to imply, but will list them for grins.
First, it implies you know that you should backup, so essentially you are an IT person. Second, it implies you actually took a backup, which implies you don’t really exist. Third, it implies you took a backup recently, and now we’re pushing the limits of quantum theory. Fourth, it implies your backup knows about permissions (not that hard to believe) and fifth that it wasn’t lost in the same operation that messed up your permissions.
This article is primarily made for when you mess up application files permissions, like those that actually run the software (OpenOffice, Pidgin, Thunderbird, Firefox, etc.). These permissions are set to typically be world-usable but not world-writable. Most of the time (with SUSE) they are installed via RPMs and are owned by ‘root’ and while ‘root’ can tinker with the actual contents of the files most regular users can only access the files (execute, read) which keeps them safe from your own bad decisions. In those RARE cases where you become ‘root’ and are following directions on how to configure some random package and you mistype with a ‘chmod’ or ‘chown’ command you can quickly make your system unusable just like you can with any bad command in any OS. A few options are available to help us in these cases. The RPM software has been around for quite a while and provides options out of the box… yes, without third-party software, Linux Just Does That.
So let’s dive in. For those of you not inclined to use the command line, welcome to computers. Not just microsoft computers, but real computers. If you hadn’t been tinkering as ‘root’ in the first place this might not be a problem now but the only way to efficiently troubleshoot any system (including windows or Mac, in my opinion) is from the command line. I’ll explain why in another article someday, but for now I’ll just be very clear in what needs to be done.
The ‘rpm’ command is the interface to the RPM database which holds information about files/directories and tons of metadata about your application and other files. The command has a lot of options to use, but a few (when learned) provide the bulk of everything necessary (use ‘man’ pages for the rest when needed of course).
A few of the options you should know include -q (query), -V (verify), -i (install), -U (upgrade), -F (freshen), and two new ones that I just learned, –setperms –setugids (set permissions and set ownership, respectively). So how do you use these?
First, the -q option: this is the basic starting point for RPM if you already have your system installed. It basically lets you query the RPM database (store of information about installed stuff) or a package on your machine for information. This option is combined with other flags to get more data and in this it can be confusing. For example there is a -i parameter that lets you install packages, but if you use -qi it gets information about whatever you are querying. I’ll provide some examples for clarification.
rpm -qa #Query All, as in everything in your RPM database. By default this dumps package names to the screen ab@mybox:~/Desktop> rpm -qa | head -3 terminfo-base-5.6-90.55 libwnck-lang-2.24.1-2.22 libgnome-lang-2.24.1-2.28
Notice that I selected three results and they happened to be just three packages from my system retrieved in who-knows-what order. The last one looks to be related to Gnome, which makes sense since I use it primarily. If I select that one specifically and use the -qi (query information) options I get the following:
ab@mybox:~/Desktop> rpm -qi libgnome-lang Name : libgnome-lang Relocations: (not relocatable) Version : 2.24.1 Vendor: SUSE LINUX Products GmbH, Nuernberg, Germany Release : 2.28 Build Date: Wed 25 Feb 2009 03:41:23 AM MST Install Date: Fri 06 Mar 2009 11:37:04 PM MST Build Host: baur Group : System/Localization Source RPM: libgnome-2.24.1-2.28.src.rpm Size : 2565938 License: GPL v2 or later; LGPL v2.1 or later Signature : RSA/8, Wed 25 Feb 2009 03:41:54 AM MST, Key ID e3a5c360307e3d54 Packager : http://bugs.opensuse.org URL : http://www.gnome.org/ Summary : Languages for package libgnome Description : Provides translations to the package libgnome Distribution: SUSE Linux Enterprise 11
Okay, now this is getting interesting. I just found that my box is SLE(D) 11 and this is a ligbnome-lang package version 2.24.1, built on some box named ‘baur’ from my vendor (SUSE) in Germany (where SUSE is from). I see its size, a signature (so I know it’s really what it says it is), a license, and all kinds of good information. Let’s go a step further and I can query all of the files that the RPM database knows about that came from this package. The ‘-ql’ option lets me List all files from this package:
rpm -ql libgnome-lang |head -3 /usr/share/locale/am/LC_MESSAGES/libgnome-2.0.mo /usr/share/locale/ar/LC_MESSAGES/libgnome-2.0.mo /usr/share/locale/as/LC_MESSAGES/libgnome-2.0.mo
Okay, now I can see three (of however-man) files that were laid down by this package. Where are we going with all of this? Well the RPM database can be (and often is) told to keep track of metadata about these files, such as ownerships, permissions, checksums, etc. Let’s try the -V (verify) option with the command:
rpm -V libgnome-lang
After a short delay I’m right back to my prompt with no output. This means that nothing was found to be different. You could add a lower-case ‘v’ to that in order to get more-verbose output and see a bunch of lines like this if desired, or you could just `echo $?` and see the return code from the command was zero (0) which typically means everything is good regardless of the command used:
........ /usr/share/locale/am/LC_MESSAGES/libgnome-2.0.mo ........ /usr/share/locale/ar/LC_MESSAGES/libgnome-2.0.mo ........ /usr/share/locale/as/LC_MESSAGES/libgnome-2.0.mo ........ /usr/share/locale/az/LC_MESSAGES/libgnome-2.0.mo
Notice the leading dots? They would be replaced by characters indicating a problem if there was a discrepancy detected. For example configuration files are often changed by users or the system or something and they therefore often show up as having a different size or md5 checksum from the original, but this is expected…. they’re configuration files. As an example I know something has modified my /etc/profile file incorrectly (no, you shouldn’t be modifying this directly… create your own user-level scripts in ~/.profile or else create your own system-wide scripts in the /etc/profile.d directory) so let’s see which package owns it:
rpm -qf /etc/profile aaa_base-11-6.3
The result is that aaa_base owns this file. If I were to get a new patch to aaa_base tomorrow my custom changes would be lost when the files were updated. Let’s run the Verify option against aaa_base:
rpm -V aaa_base .......T c /etc/csh.login S.5....T c /etc/inittab S.5....T c /etc/mailcap S.5....T c /etc/mime.types S.5....T c /etc/profile
So here we can see that the Size, the md5sum, and the Timestamp are changed on several files, and the Timestamp is changed alone on /etc/csh.login. How were they changed? Who knows, but I know where they originated and now I know that something in them may be wrong. If I were troubleshooting an issue and found that some important files, perhaps libraries or executables (binaries) were changed then I would be very suspicious. Configuration files change all the time, but binary files should not unless the package changes (typically).
Since permissions were how I led into all of this let’s pretend that those are wrong. I’ll demo that by hacking my permissions badly on purpose:
chmod 777 /etc/profile rpm -V aaa_base .......T c /etc/csh.login S.5....T c /etc/inittab S.5....T c /etc/mailcap S.5....T c /etc/mime.types SM5....T c /etc/profile
Notice that now my Mode (permissions) are wrong. Currently they are set to any average user can come in here and add anything they want to my default profile, which could include a bad command like ‘rm’ with recursive and force flags (nuking my hard drive quickly the next time I login as ‘root’ for some bad reason). I need to fix this but I just don’t know what the permissions used to be (I really don’t… I should have backed this up…. if my demo fails then this article is going to be a really costly one to me). The options to set these back to what the RPM database knows are –setperms and –setugids. In my example I have the wrong permissions on this one file so I’ll have RPM set all of the permissions for the files in that package back for me:
rpm --setperms aaa_base chmod: cannot access `/var/adm/fillup-templates/gshadow.aaa_base': No such file or directory
Now this is a good example for two reasons. First, my default bash prompt tells me the return code for my previous command and it is ‘0’ so all is well, but the warning above is interesting. Verify your own with `echo $?` before you do anything else and you should get a ‘0’ back if everything went properly. What does that message above about gshadow.aaa_base mean? Well apparently that file (/var/adm/fillup-templates/gshadow.aaa_base) exists per this package, but it couldn’t be found. That’s not really related to my initial problem, though, so let’s run the Verify again:
root@mybox:~# rpm -V aaa_base .......T c /etc/csh.login S.5....T c /etc/inittab S.5....T c /etc/mailcap S.5....T c /etc/mime.types S.5....T c /etc/profile
Look at that… the ‘M’ (mode) is now gone because it matches what the RPM database has recorded for it. So doing this for any packages that I know I have broken lets me put permissions back to normal. We saw that there is a –setugids option that fixes ownership so let’s break that next:
chown ab:wheel /etc/profile root@mybox:~# rpm -V aaa_base .......T c /etc/csh.login S.5....T c /etc/inittab S.5....T c /etc/mailcap S.5....T c /etc/mime.types S.5..UGT c /etc/profile
So now we see that the User and Group ownership options are wrong for my /etc/profile file. It’s owned by my user (‘ab’) and the ‘wheel’ group. According to my assumption I should be able to run one command to fix this nicely for me:
rpm --setugids aaa_base chown: cannot access `/var/adm/fillup-templates/gshadow.aaa_base': No such file or directory chgrp: cannot access `/var/adm/fillup-templates/gshadow.aaa_base': No such file or directory root@mybox:~# rpm -V aaa_base .......T c /etc/csh.login S.5....T c /etc/inittab S.5....T c /etc/mailcap S.5....T c /etc/mime.types S.5....T c /etc/profile root@mybox:~# ll /etc/profile -rw-r--r-- 1 root root 9663 Mar 25 21:28 /etc/profile
Voila, the ownerships are back to ‘root’ (user) and ‘root’ (group) and the Verify comes back clean(er) again.
So what does this all really mean? It’s possible, because Linux Just Does That, to fix permissions, ownerships, and other things recorded by the RPM database (and stored there from the package that was used for the installation) out of the box. The system keeps track of these for you (among who knows how many other things) so it can verify the integrity of applications that came in this way. Now there are a few caveats to note. I believe (though have not confirmed) that the builder of the RPM package must specify which of the options (Size, Mode, md5, User-owner, Group-owner, Timestamp, etc.) are saved by the package when the package is actually built. This is a developer thing that you as an end user don’t need to worry about since I’ve seen very few packages NOT store all of these options on all files that were within the package but keep it in mind. If I were to create a package and didn’t save these data as I built the package then the Verify option of RPM wouldn’t be as useful to you. Also, as we mentioned, configuration files change size/contents all the time since that is their purpose.
Also it is possible (and regularly happens) that files are not owned by a specific RPM even though that RPM may have created them. This usually applies to configuration files and not more-useful (in this case) files like libraries and other binaries or application files. As an end user I create documents all the time as well that obviously don’t come from any given RPM. /etc/passwd is a critical file that fits this profile on my system; it is not owned by any package but it had better be there and had better have the right permissions. As a configuration file it always changes from time to time so keep this in mind.
Finally even applications can be shipped without an RPM package. For example I use Thunderbird regularly and I just extract it to an ‘apps’ directory under my user’s home directory so it is only accessible by my user. Since it didn’t come with an RPM there is no way to do the tests above (any of them) for this package. That’s my own prerogative and it’s something to be aware of since many great applications may not come in RPM format. Anything you get via the SUSE repositories (or similar for other RPM-based distributions) should be fine since they are based around the concepts of packages.
That leads to another topic: what about other distributions and operating systems? RPM has had this functionality for years but there are others… Debian and Ubuntu use .deb packages. Gentoo uses a tool called ’emerge’ to build its software. Solaris uses PKG commands (pkginfo, for example) and its own type of packages. There are a lot of package options out there and this is all fairly common stuff for package management but I do not know the details for other systems. Fedora (another RPM-based distribution) obviously has it, and a site I found doing research on this led me to some the pkgchk command on Solaris: http://www.cyberciti.biz/tips/reset-rhel-centos-fedora-package-file-permission.html. A bit more use of Google revealed ‘debsums’ which appears to be available for Debian-based distributions (Debian, Ubuntu, Backtrack 4, etc.).
So what can we learn from all of this? Well, a few things come to mind. First, this is troubleshooting and fixing issues that you know are broken. I wouldn’t typically recommend running something like the following to magically go and reset permissions on all files on your system that come from packages. Will it hurt things? Maybe not, but it’s not a good idea (just like anything as ‘root’ without being very careful and understanding the consequences is not a good idea):
rpm -qa | xargs rpm --setugids --setperms #No seriously, don't run this unless you're ready to rebuild your box.
With that said we can use these skills to fix things. When you try to SSH to a machine you just mangled unknowingly and find you cannot login because the SSH server is implementing some decent security you can fix files using the RPM command. You can also reinstall software of course but sometimes that is not the best option; using a chisel is always best even if there is a jackhammer or sledgehammer nearby assuming the chisel does the job (feel free to ask any archaeologist).
While the information here is perhaps a bit complex it’s meant to illustrate a point more than to help just anybody start messing with permissions. Troubleshooting is a skill. Tracing a symptom to a cause is a skill. Hopefully with a bit of the information here and some time and effort a system that you otherwise think is lost can be recovered and restored to full health. Reinstalling software shouldn’t be the norm… it should be the exception.
Another point I find myself making over and over is to check other references. I mentioned the ‘man’ pages above; in case you are not aware there is a command in Linux/Unix called ‘man’ that accesses a built-in manual of data (there is also ‘info’ that gives a different manual). If I wanted to know more about the ‘rpm’ command I would type the following on any machine with the ‘rpm’ command installed and could likely get more than my heart’s content about the command, its options, its development history, and whatever else the developers threw in there:
‘man’ itself is a command on the most (if not all) machines. Another option to find out more about a given command is to append ‘–help’ as its first parameter:
The following section results from that command being executed and this parameter (or a variant of it) works for every command I have used:
ab@mybox:~/Desktop> rpm --help Usage: rpm [OPTION...] --quiet Query options (with -q or --query): -c, --configfiles list all configuration files -d, --docfiles list all documentation files --dump dump basic file information -l, --list list files in package -P, --patches list patches or patched files --queryformat=QUERYFORMAT use the following query format -s, --state display the states of the listed files -a, --all query/verify all packages -f, --file query/verify package(s) owning file -g, --group query/verify package(s) in group -p, --package query/verify a package file -W, --ftswalk query/verify package(s) from TOP file tree walk --pkgid query/verify package(s) with package identifier --hdrid query/verify package(s) with header identifier --fileid query/verify package(s) with file identifier --specfile query a spec file --triggeredby query the package(s) triggered by the package --whatrequires query/verify the package(s) which require a dependency --whatprovides query/verify the package(s) which provide a dependency --nomanifest do not process non-package files as manifests Verify options (with -V or --verify): --nomd5 don't verify MD5 digest of files --nofiles don't verify files in package --nodeps don't verify package dependencies --noscript don't execute verify script(s) -a, --all query/verify all packages -f, --file query/verify package(s) owning file -g, --group query/verify package(s) in group -p, --package query/verify a package file -W, --ftswalk query/verify package(s) from TOP file tree walk --pkgid query/verify package(s) with package identifier --hdrid query/verify package(s) with header identifier --fileid query/verify package(s) with file identifier --specfile query a spec file --triggeredby query the package(s) triggered by the package --whatrequires query/verify the package(s) which require a dependency --whatprovides query/verify the package(s) which provide a dependency --nomanifest do not process non-package files as manifests File tree walk options (with --ftswalk): --comfollow FTS_COMFOLLOW: follow command line symlinks --logical FTS_LOGICAL: logical walk --nochdir FTS_NOCHDIR: don't change directories --nostat FTS_NOSTAT: don't get stat info --physical FTS_PHYSICAL: physical walk --seedot FTS_SEEDOT: return dot and dot-dot --xdev FTS_XDEV: don't cross devices --whiteout FTS_WHITEOUT: return whiteout information Signature options: --addsign sign package(s) (identical to --resign) -K, --checksig verify package signature(s) --delsign delete package signatures --import import an armored public key --resign sign package(s) (identical to --addsign) --nodigest don't verify package digest(s) --nosignature don't verify package signature(s) Database options: --initdb initialize database --rebuilddb rebuild database inverted lists from installed package headers Install/Upgrade/Erase options: --aid add suggested packages to transaction --allfiles install all files, even configurations which might otherwise be skipped --allmatches remove all packages which match <package> (normally an error is generated if <package> specified multiple packages) --badreloc relocate files in non-relocatable package -e, --erase=<package>+ erase (uninstall) package --excludedocs do not install documentation --excludepath=<path> skip files with leading component <path> --fileconflicts detect file conflicts between packages --force short hand for --replacepkgs --replacefiles -F, --freshen=<packagefile>+ upgrade package(s) if already installed -h, --hash print hash marks as package installs (good with -v) --ignorearch don't verify package architecture --ignoreos don't verify package operating system --ignoresize don't check disk space before installing -i, --install install package(s) --justdb update the database, but do not modify the filesystem --nodeps do not verify package dependencies --nomd5 don't verify MD5 digest of files --nocontexts don't install file security contexts --noorder do not reorder package installation to satisfy dependencies --nosuggest do not suggest missing dependency resolution(s) --noscripts do not execute package scriptlet(s) --notriggers do not execute any scriptlet(s) triggered by this package --oldpackage upgrade to an old version of the package (--force on upgrades does this automatically) --percent print percentages as package installs --prefix=<dir> relocate the package to <dir>, if relocatable --relocate=<old>=<new> relocate files from path <old> to <new> --repackage save erased package files by repackaging --replacefiles ignore file conflicts between packages --replacepkgs reinstall if the package is already present --test don't install, but tell if it would work or not -U, --upgrade=<packagefile>+ upgrade package(s) Common options for all rpm modes and executables: -D, --define='MACRO EXPR' define MACRO with value EXPR -E, --eval='EXPR' print macro expansion of EXPR --macros=<FILE:...> read <FILE:...> instead of default file(s) --nodigest don't verify package digest(s) --nosignature don't verify package signature(s) --rcfile=<FILE:...> read <FILE:...> instead of default file(s) -r, --root=ROOT use ROOT as top level directory (default: "/") --querytags display known query tags --showrc display final rpmrc and macro configuration --quiet provide less detailed output -v, --verbose provide more detailed output --version print the version of rpm being used Options implemented via popt alias/exec: --scripts list install/erase scriptlets from package(s) --setperms set permissions of files in a package --setugids set user/group ownership of files in a package --conflicts list capabilities this package conflicts with --obsoletes list other packages removed by installing this package --provides list capabilities that this package provides --requires list capabilities required by package(s) --suggests list capabilities this package suggests --recommends list capabilities this package recommends --enhances list capabilities this package enhances --supplements list capabilities this package supplements --basedon list packages this patch-rpm is based on --info list descriptive information from package(s) --changelog list change logs for this package --xml list metadata in xml --triggers list trigger scriptlets from package(s) --last list package(s) by install time, most recent first --dupes list duplicated packages --filesbypkg list all files from each package --fileclass list file names with classes --filecolor list file names with colors --filecontext list file names with security context from header --fscontext list file names with security context from file system --recontext list file names with security context from policy RE --fileprovide list file names with provides --filerequire list file names with requires --buildpolicy=<policy> set buildroot <policy> (e.g. compress man pages) --with=<option> enable configure <option> for build --without=<option> disable configure <option> for build Help options: -?, --help Show this help message --usage Display brief usage message
The resources for most commands are plentiful in the computing world. Forums, IRC, product-specific e-mail lists, etc. all focus on helping with learning, and most of it is free.