Python2.6, Trac0.11.x, mod_wsgi and mod_python on RHEL5.x

New server and new problems.

Late last year my patched along circa 1999 Dell server decided to go belly up. That server was running Centos-5.2 and hosted my Trac installation running Trac-0.11b, running in Apache using mod_wsgi. So recently we finally replaced the server with a brand spanking new Dell server (heck if its predecessor lasted almost 10years, I hope this new one lasts at least 5years). I installed RHEL5.3 on this server instead of Centos since we (the IT group at work) want to simplify our OS variations and stick with the commercially available RedHat Enterprise Linux. And that is where my problems began.....

Well as we all know RHEL5.x still only includes Python2.4 with the distribution. Centos and other variant have started to include or at least are upgradable to Python2.5. While RHEL5 does include mod_python, mod_wsgi, Python2.4 and Trac0.10.x... I needed Trac0.11.x

Solution: drop RHEL and go with Centos... just kidding

So actually it turns out not to be that bad... you just have to do a lot of manual compliations... so here is my step by step solution:

Step 1. install Python from source, I went with the latest 2.6.1 (not 3 yet since on my Mac I've had backwards compitibility issues and didn't want to deal with them)

./configure --with-threads --enable-shared
make
make altinstall
ldconfig

the 2 key steps: --enable-shared in the configure statement does the work needed later by mod_wsgi and mod_python, and 'make altinstall'  which tells the installation to use /usr/local and to appended the 2.6 to the commands to prevent messing up system scripts which are based on Python 2.4 and in many cases (yum included) do not work with Pytohn2.6. The last step 'ldconfig' simply refreshes the library information which is needed before compiling mod_wsgi or mod_python from source.

optionally required step: if the shared library for Python is not seen in the '.../python2.6/config/' you'll need to add a symbolic link

ln -s ../../libpython2.5.so .

now on my server, the python config directory was at:  /usr/local/lib/python2.6/config

Step 2. install 'mod_wsgi' from source.

./configure --with-python=/usr/local/bin/python2.6
make
make install

here the only statement of difference is to tell mod_wsgi to use python2.6 instead of the system default 2.4, now if you have installation errors check out http://code.google.com/p/modwsgi/wiki/InstallationIssues this page also explains the optional step after the Python installation.


Step 3. install 'mod_python' from source.

./configure --with-python=/usr/local/bin/python2.6
make
make install

as you can see, there is no difference between the methods for compiling mod_python and mod_wsgi. And again the main reason for compiling from source is to enable the use of Python2.6. And since both of these modules will be using Python, this is the reason of compiling Python2.6 with the '--enable-shared' flag.

Step 4. download and install the python setuptools for python 2.6 using the py26.egg.


Step 5.  using easy_install tool from the setuptools in Step 4, install Trac.

easy_install-2.6 trac

this will automatically install Trac.11.x since it is the latest version compatible with Python2.6, and using 'easy_install-2.6' will ensure that you are using Python 2.6

Step 6. configure Apache like instructed by TracModWSGI.

Now that wasn't to painful, the only clevat... now since all these packages have been installed from source, for security purposes, you have to remember on your own to update/patch these installs.

update: I've run into a slight problem where Trac can't access SVN since the libraries aren't 'accessible' by Python2.6, I'l update again when this is solved

update2:  Solution to get the SVN-SWIG installation to work with Python2.6... recompile subversion, only 'change' needed was manually hard coding the version of Python in the './configure' file

# Python: Used for testsuite, and bindings

#PYTHON="`$abs_srcdir/build/find_python.sh`"
PYTHON="/usr/local/bin/python2.6"

 Only remaining issue was where the files were installed to. Presumingly there is another line in the ./configure file to correct this, but I went the 'quick' way... symbolic links.

cd /usr/local/lib/python2.6
ln -s /usr/local/lib/svn-python/* .

Postgre and Sqlite play nice

At work we use a Postgre database as our primary datastore. At the same time I maintain several flat-file databases (which are basically csv files) for external processing. The big problem is that the data in my flat-file database and the Postgre database are not always in sync. The data in the database can be modified/updated via our flagship application (H*Wind) but the data in my files has to be updated manually.... as you can guess that means it only happens about 1-2 times a year (if that)....

Anyways... one of the tasks I'm trying to complete before the start of the next Hurricane season is to rewrite data collection suite of C programs into a nice 'simple' Python package. Being that my C programs depend on the flat-files (which as I already mentioned can be out of date) I want to remove them... well I want to remove the Hashtable step. Currently any of my programs that depend on the flat-files, has to first read the file and convert it to a big hashtable. This is great when it comes to accessing the data, but when the program has to do this every time it runs and considering the fact that is runs several times an hour.... well you can guess what I'm getting at. So what I want to do is access the database since this is what databases are design for... but I don't want to have to depend on the network connection being alive while I'm developing my Python package, so in steps Sqlite3. I know I could install Postgre and copy the desired tables to it, but that seems a bit overkill when all I want is a single table that has only a few 1000 entries.

The minute I started development, I was already thinking Sqlite, but initially I was thinking of dumping my csv files into the Sqlite database... problem, my csv files are not 'simple'. So the solution.... PG_DUMP! That is right... while I was looking through the Sqlite documentation I came across this:

http://www.sqlite.org/cvstrac/wiki?p=ConverterTools

And in the section where it discussed Postgre, it gives a quick short set of steps to convert the data, and yes it works...

• PostgreSQL to SQLite3 After a little experimenting, I found a way to import data from PostgreSQL into SQLite, version 3, without having non-ASCII characters ruin things (something that getting pg_dump to export SQL statements did...). It could be enormously improved with a little scripting love, but as I only had one database with a couple of big tables to deal the below was fine, however... post improvements!

Export the data from your database as tab delimited text. Create one text file for every table in your database:

$ pg_dump -a  -t  > /tmp/
    ...and so on...

Trim off crap from header and footer from each file, eg:

$ nano -w /tmp/

If required, create SQLite file and tables corresponding to tables in the original PostgreSQL database:

$ sqlite3 
    sqlite> CREATE TABLE ...etc...

Finally, import the dump files into SQLite, remembering to change the default import separator to a :

$ sqlite3 
    sqlite> .separator "\t"
    sqlite> .import  

So now all I have to do next is learn how to interact with Sqlite from within Python.... maybe not to you, but for me, this is fun!

Playing with Wordpress and Python and the metaWeblog API

For those who know me, they know I work for the Hurricane Research Division and the Surface Wind Analysis group (http://hurricanewinds.noaa.gov). Well anyways we've been wanting to move forward with a new product site, something more up to date with the status of the web and the world. Anyways initially my plan was to customize Trac, which we use as our project management package, as the 'engine' for controlling our new website. I saw the built-in Wiki tools and RSS feeds as the driving force. But the more I looked at it, I began to think that I was going about it the hard way. One of the goals in my mind was to have an RSS feed for our analysis and with the availability of a variety of blog tools, I knew there had to be a better way.

Anyways weekend before last with Hurricane Gustav, we found that the building FTP server was getting overloaded and denying access to many people who needed it, this also prevented many people from gaining access to our analysis, especially a number of Emergency Management people. I right away started looking at an alternative.

Since I was already looking for alternative solutions to my Trac idea, my first thought was to setup some kind blog. Being that I'm a Blogger user (as you can tell by this blog), I first looked at Blogger. My first thought was to setup an automatic email solution posting to Blogger. Try as a might using mail and other CLI mail tools for Linux, I couldn't seen to figure out how to get my attachments sent and actually present a post with some sort of decent layout. Next I looked at Wordpress.com, but they didn't have a email option and I didn't think it was going to be straight forward to send images to the blog.... Well by this time Gustav had already made its landfall along the Louisiana coast, so I said forget it and went on.

Well that didn't last long... I looked out on the horizon and new that Ike was going to be just as much trouble as Gustav was when it came to internet traffic. I knew I needed to find that solution. So back to the blog engines I looked. This time I went a different route... when I first started with the idea of customizing Trac, it already meant that I was going to have to host my own site, so why not now. So back to WordPress. This time to Wordpress.org and its 5-minute installation guide. They weren't kidding, in 5 minutes I was up and running a basic Wordpress blog. But so what... I still needed to figure out how to post to my new blog and still I needed to figure out how best to post my images...

Well for the images I cheated.... I'm not posting them to the actual blog,  I decided to simply store them at location accessible by WordPress (basically in another directory on my webserver). So that solves that problem, now all I need to do is create a blog posting tool that sends a nicely formatted blog post with links and standard img tags. No problem... in steps my new favorite language Python, which I started working with back in March (other posts).

I already knew that Python had lots of built-in default packages for working with the Web, heck, Trac is completely written in Python, what I didn't realize was how easy it was to work with. One of the built-in modules is an XML-RPC module which is all you need to start writing a Python program to post to WordPress (or most blog engines for that matter). The XML-RPC module can work with most of the blogging APIs including MovableType and the MetaWeblog API. For WordPress I went with the MetaWeblog API, WordPress implements MovableType API and the Blogger API, the MetaWeblog API was in my mind more generic and most likely more useful for me later down the line.

Anyways when I started writing this post I was planning on including snippets of my Python code but as since this post has gotten to long, I think the snippets would get lost and not be of much help here, so I'll write a second post that deals strictly with using Python to write to a WordPress blog.

So to get to the conclusion, check out http://storm.aoml.noaa.gov/wordpress to see the setup. I chose the Blocks2 theme from by mg12 and I also installed a PDA/iPhone plugin for mobile browsing. Plugins is where I find the big power of WordPress falls. There are so many plugins available that basically there is a plugin for everything. I'm sure other 'hostable' blog engines like MovableType and such have as many plugins, but for now I'm sticking with WordPress. I've got it setup and since it works, I don't think I'll be looking at the others for some time... or at least until I get bored with my current setup.... Oh one other plus for WordPress, its FREE!

As for this blog... I think it will remain a Blogger blog. For now Blogger works for me... I do however see myself switching to Wordpress eventually, but not until I finally choose a domain name and actually break down and get a hosted solution. WordPress is great, but the free version available at WordPress.com is just to much of a tease when you know what WordPress can do and how easy it is to setup.

As for my code snippets... I'll post that tomorrow.

pointless dumb bugs...

As I briefly mentioned yesterday in my post about 'Eclipse, Subversion and Mac OSX' along with my SVN issues on my Mac I was having SVN issues on my Centos 5 box relating to Python and the setuptools. Well while thinking it was an SVN issue help me solve my Mac problem, it actually had nothing to do with my Centos/easy_install problem. Basically after I finished fixing my Mac/Eclipse issue I took the time to actually read the error I was recieving when installing 'eggs' via easy_install. The error actually was a simple Python error.

error: global 'log' not declared.

sorry I don't remember the exact error and since I've fixed it, I don't exactly see it any more

Anyways... so it turns out the Python imports were not being passed between packages 'correctly', or it just wasn't working on my server... anyways the simple fix was to add the following line to the file/package in question:

from distutils import log

 the file on my end was '/usr/lib/python2.4/site-packages/setuptools/command/sdist.py'. It is probably an error in other packages too, but as since I haven't run into them yet.... All I hope is that it gets fixed with future updates, we'll see what happens.

Python make pretty pictures

Well as those who know me, know that I've been on sort of a Python kick lately. A few months ago I started learning Python. So now that a few months have past I'll share the latest reason I like Python... Graphics....

Early in my time with HRD I started working with graphical products. Initially I was playing with a product called IDL. I thought the product was great, but we had a mathematician in join our group who was way more advanced with IDL than I and I let him take control of the product and was very happy to move on to other projects. This said I've always been interested in computer graphics as far back as I can remember... in grade school it started with Logo, and then Graphics on an Apple IIe.... Back to the some what present... even in grad school I took a course on OpenGL, even made a nifty 3d maze game. Anyways again I digress (pretty typical of all my blog posts...)

So graphics has always been something I've wanted to get into, plus with the always evident idea in my group of making our product portable and deployable, I also new one day that I'd have to find a non-IDL/commercial solution. I state non-commercial as well since I could also have looked at MatLab or such. Anyways the first two products that leapt out at me a while back were GDL a GNU clone of IDL and Octave, another GNU project compatible with MatLab. The only problem was that GDL at the time did not support many of the IDL tools we were now using, and Octave... well I still don't know MatLab so that wasn't going to work.

Anyways.... so I place the idea of a free and distributable version of our product tools on the very back burner... you know the one that was never even attached to the stove...

So we move foward in time to the real present... Like I said in the beginning of the post, a few months ago I started learning Python. Like learning any new programming language, it is mostly syntax and package/module discovery. The syntax part was straight forwards, I'm a C/Fortran/Java/Unix scripter.... so yes, Python syntax, not anything new. As for packages... well they are abuyndant, but easily managable. Unlike Perl which I don't even know where to start on how large it has gotten (and yes in my early days I was even a Perl programmer), trying to find the 'best' package can take as long as trying to write your own... So as for Python, I quickly learned to read and write ASCII files and perform some simple mathematical calculations on data and a few other simpler tasks.

Then a co-worker of mine asked me about writing a program for a project he was looking into. The project seemed straight forward enough... read in a couple years worth of data (all of which were simple 2d matrixs), perform some simple stats on them (mean, std, median, min, max), spit out a simple ASCII summary, and generate some pretty pictures. All of the first steps seemed straight forward, the pretty pictures... well that was a different story. Anyways still being on my Pytohn kick, I quickly started out on this new project in Python. Took a couple of days to get the files read and stats processed, was even easier when I learned about NumPy and SciPy. NumPy is great, it provides great array handling for large matrixes and such. Plus now the reason for this post... an additional package called MatPlotLib is also part of the NumPy/SciPy family. MatPlotLib made plotting the matrixes an almost breeze. Basically once I figured out my dimensions and the geospatial area for my map to plot on, all I had to do was give it my NumPy array and say draw, and poof.... pretty picture.

Don't ask me what the picture is exactly of, but ain't it pretty? I do know that it is some sort of thermal data from the GOES satellite (or atleast I think it is from the GOES satellite). But ain't that cool... all I did was create Basemap and then tell it to plot using the pcolor(...) command. It was that simple. Another reason I think Python and I are going to have a long happy relationship. A few more tweaks and I should have my program fully ready to go... only issue is... currently it is kind of slow.

When White spaces aren't White

As most know, most of my programming work relates to data processing. One of my responsibilities with the Hurricane Research Division and the H*Wind team to be exact has been to process data and insert it in our database. So over the years (almost 12) I've written a number of programs to help along my way. A few years ago (before I discovered Python), I wrote a rather significant program for my own use. It is a generic parser. I wrote it as part of a project to re-analyze Hurricane Katrina. We had data coming in from all over the place and almost none of it in the same 'format', format in quotes since I refer to column order, field delimiters, units, etc. but almost all the data was ASCII. Anyways... I'm digressing from my point... so this parser was basically designed so that the user (basically me) would simply list the column order and such, and out would pop a file ready for the database. Well to the point of the blog post title, today (and yesterday) I was trying to figure out why my program wasn't working. The default delimiter in my program is WHITESPACE, basically any not alpha-numeric-symbol character, aka space, tab, return.... well incomes this file I processing... it was generated in Fortran, so my first assumption is that it should be space delimited, but it was not, it was TAB delimited. Okay no problem, my program can do that, but it didn't. For some reason, TAB is not being recognized as a whitespace.... argh, however when I distinctly tell my program with command-line arguments to accept TAB as a delimiter, it works.... argh..... Oh well, at least it works... thankfully. I sure the issue has something to do with being prepared on a HPUX machine, transfered to a Linux box, emailed to a Mac box and then processed.... you know somewhere in there something had to get lost in whitespace translation.... anyways... again the program works, but it is always fun when you write a program to do something and it doesn't work the way it should... thank goodness for Unix and the many workarounds that are available.

Of course this whole experience is another reason I'm really starting to like Python programming... maybe I should take the time to re-write this c program into Python... then again it works 99% of the time, so if it is not broke, why fix it?

Learning Python

I'm a long time C developer. Basically I figure that anything that needed to be coded could be coded in C. I still hold that belief, but 7-8 years ago I came to the realization that while that may be true, it is not always the best answer. For project reasons at work I learned Fortran and began to recognize its value when it comes to math. For other project reasons I learn how powerful simple Unix scripts can be. So basically for the last 7-8 years I've seen Unix scripting as a steering tool, C for text processing, Fortran for math, and Java for object-oriented and gui tasks. But recently based on a number of article from Computing in Science and Engineering (CiSE) I've come to learn about Python.

Python is a scripting language at its core, but overall it is much more. Python has a variety of modules and add-ons. Some basic include modules include NumPy/SciPy for math and science, CSV for delimited text, and XML for xml parsing and generation. My first Python program took me only a day or so to develop and test, it was a CSV translator. Basically it converted one CSV file into another. My next program took me another 2-3 days and it converted a gridded file into a CSV formatted file. And today, I wrote my first XML parser. Using 'minidom' from the xml package I was able to put together a simple parser for converting an XML file into a csv file.

All this I've basically accomplished over a few weeks of learning using only a handful of simple resources both online and in print. Yes I have many years of experience with a variety of programming languages, but I definitely have found Python the easiest to learn yet. Here is a quick list of resources I've found helpful:

• Tutorial at Python.org
• Python.org Library Reference
• Dive into Python by Mark Pilgrim
• Python/XML Howto by A.M. Kuchling
• Python XML Frequently Asked Questions by Dave Kuhlman
• Book: XML Processing with Perl, Python, and PHP by Martin C. Brown
• Magazine: CiSE issue: Python: Batteries Included Volume 9, Number 3
• Python and XML: An Introduction
  
• Creating XML with Python

I've bookmarked a number of other sites for future use but have not gotten to them yet. My next task will be to add a simple XML version of the gridded data file mentioned above, plus add NetCDF writing support and GIS writting support, namely ESRI Shapefiles. I also plan on learning howto wrap my C packages with Python binding. To put it plainly, things that I've spent years developing and testing in C for parsing weather related data, will all slowly be phased out and replaced with python code and modules and all opensource.

update 2008-03-19: added 2 more links