Whitesquare Software

I’ve run into some limitations of the EnVers addon to Hibernate that required me to dig into the Hibernate source code and look at pushing up some changes. Luckily the Hibernate Core is now hosted on github so this makes it easy to fork and post my updates.

The discussion on my changes for EnVers are outlined in this discussion.

So first step was to create my own fork of hibernate-core and then create a branch from the 4.0.0.Final tag on the hibernate-core repo.

Then cloning the repo to my local machine was as simple as:

git clone https://stevemac007@github.com/stevemac007/hibernate-core.git


A few simple commits later and I had a working solution.

Then my first git gotcha. I made a change and did a local commit, and then pushed the change upstream. But I had missed a file.
That’s OK from what I know you CAN amend the previous commit, and so I did. It committed locally fine, but when I tried to push the change back upstream it failed.

I’m not sure how I need to get out of this sticky situation so I have just deleted my local clone and re-cloned the repository.

After the fact I found that I might have been able to do this to fix my problem.

I’m going to create a new test branch and see how I can fix this and will update back here how to I go.

Just installed the new iPod app to allow me to post here. Not like it will happen more often, but at least now I can.

I think I have now standardised my server choice to ubuntu server. It seems to have everything I need available as packaged and more importantly supports version upgrades in place. (and they actually work).

The other nice thing is the introduction of a simple firewall app to manage the iptables.

Only hardcore hackers and serious sys admins remember the syntax for adding rules to iptables, and even then I’m sure they still need to read the man page once in a while.

So lets take a look a the new firewall config tool ‘ufw’. It’s probably not that new, but its new to me.

It makes it really easy to add simple rules, like
ufw allow http (allow http access)

but once you get a multi-nic’d machine it still is a bit complex to work out how to allow some traffic and not others.

This is the basic syntax to limit service access based on ip address.

ufw allow from 10.11.12.0/24 to any port 5222

Hopefully I’ll remember this note is here and check back later, and anyone else that has luck searching for this stuff I hope these comments were useful.

Utils package 1.0.7 has been released to enable the release of other new modules.

Change set to be completed shortly.

Whitesquare software would like to announce the first release of DCAF v2 (version 0.1.0).

DCAF v2 builds upon the structure available in the primary version of DCAF, but introduces the ability to have non numeric primary keys, and better support for cross database identity columns.

Not all features of DCAF have been ported to V2 as of yet, and changes are still occuring that may breal future compatibility. A related dcafg v2 project is underway and should be released shortly.

Builds of dcaf v2 are available in the Whitesquare Software Maven repository.

Crickscore was originally created to allow me to try out new technologies in a place that didn’t really matter but was still important enough that it actually had to work and would be a benefit to people.

So Crickscore evolved, and now is used full time by 2 teams with an expectation that after a few more features are added it will be opened to the general public.

Over the last couple of days, one of the teams had a photo to post to the site, which was easily placed in the tomcat directory and linked to via code in the game description.  But as soon as I was to re-deploy the application the image would be gone.

I needed to build a longer term storage system for these items, allowing for  images to be uploaded and also linked to.

This saw the creation of the Media section.  As a registered user who is assigned to a team you can now upload image files to be stored and served from the crickscore system.

Its as simple as that.  You can also just link to an image already publically available on the web by selecting the Link option.

At the moment these item’s just appear on the Media tab for the team, but future enhancements will allow people and individual games to be linked to the images.

Also there is a hope that some basic video formats will be supported, but don’t hold your breath.  That is probably better done by uploading to YouTube and linking the document back.

So there’s the new feature, but the fuss is actually on how its done, if you are inclined read on to see how this was done, and proof of how quick and flexible this new “Cloud based services” era really is.

Geeky Stuff

We’ve been looking at internet hosted services (see I didn’t say Cloud) for a few things recently, most of which focused around Amazons new AWS services.

I have been playing with S3 to upload backups of my servers to the Cloud, as well as attempting to backup all the photos on the home PC.

When I saw a need for Crickscore to handle images, I thought that this would be a good time to try out serving these parts from Amazon’s Cloud Front.

For those that haven’t heard, Amazon have just released a CDN that fronts off your S3 buckets to allow any resources stored in the bucket to be served. Read a much better description over a the official Cloud Front site.

What I wanted to do was take any media file that was uploaded to crickscore and pass it through to S3 so that I could link to it using CloudFront.

I think there are ways I can directly post items up to S3, but I want to check the size, and content first and even do so processing on it, like resizing or watermarking first before it becomes publically available.

The first thing I needed to do was to find a Java Library that would allow me to administer s3 buckets, and once I found this library there wasn’t much more.

I’m using the jets3t library, which pretty much does all the hard work.  My application framework (WAF) handles multipart form uploads, so I end up with a byte[] which I can then just push up to s3 and bang its done.

Really, it’s that simple.

Code Sample

Here’s the code that I used, taken almost directly from the examples in the jets3t library.

public static Media uploadContent(User user, String name, Team team, byte[] content, String contentType) throws S3ServiceException, MediaException {

        String awsAccessKey = SystemSettings.getValue(SystemSettingConstants.S3_ACCESSKEY);
        String awsSecretKey = SystemSettings.getValue(SystemSettingConstants.S3_SECRETKEY);
        String s3BucketName = SystemSettings.getValue(SystemSettingConstants.S3_BUCKET_NAME);
        String urlPrefix = SystemSettings.getValue(SystemSettingConstants.S3_URL_PREFIX);
        //http://content.fixturetime.com/

        AWSCredentials awsCredentials =  new AWSCredentials(awsAccessKey, awsSecretKey);

        S3Service s3Service = new RestS3Service(awsCredentials);

        S3Bucket contentBucket = s3Service.getBucket(s3BucketName);

        if (contentBucket == null) {
            contentBucket = s3Service.createBucket(s3BucketName);
        }

        validateContentType(contentType);

        String filename = null;
        while (filename == null) {
            filename = generateFileName(team, contentType); 
              if (doesFileExist(s3Service, contentBucket, filename)) { 
                  filename = null; 
              }
        }

        // Create an object containing a greeting string as input stream data.
        S3Object imageFile = new S3Object(filename);
        ByteArrayInputStream greetingIS = new ByteArrayInputStream(content);
        imageFile.setDataInputStream(greetingIS);
        imageFile.setContentLength(greetingIS.available());
        imageFile.setContentType(contentType);
        imageFile.setAcl(AccessControlList.REST_CANNED_PUBLIC_READ);

        s3Service.putObject(contentBucket, imageFile);
        Media m = Media.create(user, team, name, contentType, urlPrefix + filename);

        return m;
    }

This function is pretty self explanatory, but I’ll go through it here for completeness.

You’ll first need 3 things, your S3 access key, secret key and the bucket you want to put the content in.  I store these things in a database table called SystemSetting.

Next create a set of AWSCredentials from your access and secret keys, then instantiate  a new S3Service.

I then check for the existence of my required bucket, and if the bucket does not exist the create it.

Next comes the application code, it validates the contentType supplied is a valid type (I only support jpg/png/gif).

Then the app generates a random filename and does a check on the s3 service to see if that name already exists.

Then we’re ready to store the file.  Create a new S3Object, set its content and store it using putObject on the S3Service to post it to Amazon.

My app then stores references to the media item in the Media table.

That’s it.  I now have a URL that contains a public link to the image uploaded.

Configuring Cloud Front

The only thing that I haven’t talked about here is the configuration of the CloudFront service.  The code so far takes a byte[] and uploads it to an S3 bucket.  By setting the ACL to REST_CANNED_PUBLIC_READ that makes the file public, and so would be directly accessible from an s3 type URL anyway.

My bucket is called ‘content.fixturetime.com’ so take the example file I have uploaded 04122008130-sm.jpg it is publically available on http://content.fixturetime.com.s3.amazonaws.com/04122008130-sm.jpg. But that URL is ugly, and not using the CDN that is CloudFront.

The last step was to use the Firefox plug-in S3 Organizer to configure a ‘Distribution’.  All you need to do is setup a distribution on the bucket you need and Amazon will allocate a you a dynamic hostname.  I then just created a new DNS record for content.fixturetime.com and setup a CNAME pointer to the allocated Amazon domain, and we’re done.

The same image linked from the above image is now available on http://content.fixturetime.com/04122008130-sm.jpg. But this time is actually served from regional ‘edge’ locations by Amazon automatically.

I haven’t done any testing on this configuration yet, but it was so easy to setup that I’ll take it just for the clean URL.

So we have decided to move the subversion direction, so its time to get some infrastructure working so lets get through the initial setup.

The aim of the setup is to have the following:

• Authentication from the Windows Active Directory
• Folder permissions to control write access
• Emails sent on each commit

I’ve chosen to use Ubuntu 8.10 as the base operating system, and will assume that you can install the operating system.

Setup AD Authentication

The other main step is the AD integration, this is not that straight forward, but as long as you remember to put the domain in UPPERCASE, its not too hard.

I followed the http://tech.givemethe.net/node/18 post that goes into much detail on the required packages and the testing to get authentication working.

Basically the aim is to get the PAM subsystem authenticating off the domain, which we can then use as a base for subversion to do it authentication against.

I you don’t want to authenticate off an AD, then you can skip all of this information and just use local account authentication.

Install Packages

We need to get the subversion binaries, apache and the connecters to allow apache to do its authentication from pam, along with the apache svn module.

apt-get install subversion
apt-get install apache
apt-get install libapache2-mod-auth-pam
apt-get install libapache2-svn

Dependancy management rocks.  If you don’t have apt-get then try yum (redhat based distros) if you have to download compile and install the packages manually, then there are probably more detailed walkthroughs than I have time to write.

Setup Repository

Pretty simple, just create the location and create the repository.

mkdir /usr/local/svnroot
svnadmin create /usr/local/svnroot/

Configure Apache

When we installed the mod-auth-pam and svn modules for apache2, they should have been activated.

Check this by looking in the ‘mods-enabled’ directory.

ls -l /etc/apache2/mods-enabled/
total 0
<content trimmed>
lrwxrwxrwx 1 root root 31 2008-11-24 10:11 auth_pam.load -> ../mods-available/auth_pam.load
<content trimmed>
lrwxrwxrwx 1 root root 26 2008-11-24 10:17 dav.load -> ../mods-available/dav.load
lrwxrwxrwx 1 root root 30 2008-11-24 10:17 dav_svn.conf -> ../mods-available/dav_svn.conf
lrwxrwxrwx 1 root root 30 2008-11-24 10:17 dav_svn.load -> ../mods-available/dav_svn.load
<content trimmed>

Now that we have all the pre-requisites installed, we can start to configure the applications.

Lets first start with the apache configurations

cd /etc/apache2/sites-available/
vi default

And then put in the following section in the file.  Of course you can do things like setup a Virtual Host just for subversion, but for the testing we require installing inside the default configuration is acceptable.

<Location /svn>
    DAV svn
    AuthzSVNAccessFile /etc/svn/svnaccess.conf
    SVNPath /usr/local/svnroot
    AuthUserFile /dev/null
    AuthType Basic
    AuthName "SVN repository"
    AuthBasicAuthoritative Off
    AuthPAM_Enabled on
    Require valid-user
</Location>

So a bit of detail here, this says to mount the /svn location as a WEBDAV point, using the svn provider.

We will be setting up per-path authentication shortly, so the AuthzSVNAccessFile directive shows where to access the config file.

SVNPath is the path to the actual svn repository.

AuthUserFile is to fix a bug in apache2 due to the fact we are not using the basic authentication (see below for more details).

The rest of the auth setup is pretty standard as with any apache authentication, apart from the AuthPAM_Enabled, and you can guess what that is doing.

SVN Testing

At this point you might want to comment out the AuthzSVNAccessFile line and just test that you can access the repository and the AD pam passthrough authenitcation is working.

I did this from eclipse, and it took a couple of tweeks in my config file to get the login working.

Just ensure that you can logon locally with the domain credentials, then it should all pass through apache smoothly.

I took this time to create a /trunk /branches /releases folders that I use later for security.

Per path authentication

I’ve only just started to configure the access restriction, so see http://svnbook.red-bean.com/en/1.1/ch06s04.html 2/3’s of the way down for details on the configuration of the authentications. Future posts will go into more detail on this subject.

As I only have a single repository configured I don’t need to set repository prefixes, but the link above shows more details.

What I want to do is ensure that you can’t write to trunk, only to project based sub-directories.

[groups]
admins = steve
developers = steve, someoneelse

[/]
* = r

[/trunk]
* = r

[/trunk/project1]
* = rw

[/trunk/project2]
* = rw

[/branches]
steve = rw

[/releases]
steve = rw

This setup allows any authenticated user to write to the project1 and project2 folders, but only ‘steve’ to write to /releases and /branches

Error messages

The bulk of the error messages I got was with (as expected) the apache—>pam—>AD integration.

• (9)Bad file descriptor: Could not open password file: (null)

This was caused by a bug in apache trying to do file authentication even though we are using PAM lookups.  To remove this from the log file, just add the following to the apache config.

AuthUserFile /dev/null

• (2)No such file or directory: The URI does not contain the name of a repository.

This on was my fault, I was coping a configuration from a forum post, and it used SVNParentPath instead of what I needed SVNPath. See below for a discussion on the differences.

Sidenote: Configuring multiple repositories

If you are going to run multiple repositories on the server you can have 1 set of configuration manage all repositories.  Just create all the repositories under a single directory, and use the SVNParentPath directive to point to that folder.

For example, if you have repositories in

/usr/local/svnroot/repo1
/usr/local/svnroot/repo2
/usr/local/svnroot/repo3

With repo1, repo2 and repo3 each being seperate repositories, then use

SVNParentPath /usr/local/svnroot

and you will be able to access

http://servername/svn/repo1
http://servername/svn/repo2
http://servername/svn/repo3

with the single configuration

Setup email commits

The last step is to get the

I followed the instructions on http://help.joyent.com/index.php?pg=kb.page&id=53, but they are detailed below for completeness.

If you want to use a local mailer in your configuration then you might need to install your favourite.

apt-get install sendmail

Its then just a matter of installing the svnnotify script, and configure the post-commit hook.

perl -MCPAN -e ‘install SVN::Notify’
vi /usr/local/svnroot/hooks/post-commit

Add in the content

#!/bin/sh
REPOS="$1"
REV="$2"
/usr/local/bin/svnnotify -r $REV -C -d -H HTML::ColorDiff -p $REPOS –smtp localhost -t svn-commits@mydomain.com.au –from svn-commits@mydomain.com.au –reply-to svn-commits@mydomain.com.au

Then just save the file, ensure that the owner and permissions are correct.

chown www-data /usr/local/svnroot/hooks/post-commit
chmod u+x /usr/local/svnroot/hooks/post-commit

Email Testing

You should be able to now perform a commit and see the email sent to your specified to address, if not, check the permissions on the file are executable by your apache user.  To test that the script is configured correctly you can manually run the script and check for errors

/usr/local/svnroot/hooks/post-commit /usr/local/svnroot/ 4

Next Steps

Now that we have a working repository, that authenticates off our Active Directory, has per-path access controls and emails us on each commit we can do some testing.

Items to follow up:

• Discussion on repository layout
• Virtual Host configuration inc SSL
• ViewVC integraton
• Continuous integration
• Bug / Issue tracking integration
• AD Based Group Access
  • I don’t want all users in the AD to have access, only those in the SVN Users group

Thats all for now, post any questions in the comments.

I have been working on this for years, but still can’t get it working correctly, so this is a post to rant what I would like to see, in hopes that someone might point me in the right direction.

So what do I want:

1. Ability to specify start and end tags to search for changes.
2. Group changes on multiple files by commit (based on text and date)
3. Show file version changes
4. Output in XML for stylesheet or post processing.
5. Can specify most HEAD as to-tag to show changes so far

That seems like a small list, as I’m sure there’s more, but I’ll update when I remember them.

Date support has been in the library for a while now, but I have been recently been developing on SQLServer where I need to write to datetime fields. And it seems that the java.sql.Date is just that. Just the date. No time component.

Although DCAF exposes this field back to the developer as a java.util.Date, there is some conversion down deep that actually processes it back to its java.sql.* equivalent. So a simple change to offer a TIMESTAMP datatype along with the supported DATE type.

Checkins to come soon, after more testing and a new DCAF release on the horizon.

I have been playing with connecting up the source code repository (CVS) and my issue/bug tracking tool (Mantis). The aim of all this is some transparency in relation to what changes were related to which ite,s of development and vice versa.

See it would be good to be able to say, this file has had 3 recent commits, all in different parts, and then work out what they changes were attempting to do. I know that commit comments are supposed to do this, but some times its better to have the detailed bug report to read the overall aim.

Also, the ability to collate a change log from a set of data changes and show which bugs have been worked on, resolved, etc.

So I started looking at what configuration I could use to get this integration. I was looking for some sort of control of the bug tracking tool via the cvs commit. I know that you can do things like verify code styles, and such via the one of the cvs commit hook scripts. I have a plugin to the script that allows emails to be sent upon commit.

So I started googling for scripts that could do this, there were a few around but were either old, or didn’t support my CVS–>Mantis combination.

Then gold, a project called SCMBUG.. Exactly what I was looking for, integration between a source control management system and a bug tracking tool. Even better was that it was a generic framework, that allowed plugins for different applications.

On the list of supported SCM’s was CVS (obviously), and on the list of supports Bug tools was Mantis.. YAY.

So a few hours later, its installed and configured. Works well, infact a bit too well. I have it enforcing that ANY commit to CVS must have a bug associated to it, and that the bug must be open, and it must be assigned to me. You see all of these settings are configurable via it, and I thought I’d force myself to use it in nearly it’s harshest configuration.

The only problem is that I now realise that I do a lot of random commits, fixing things that technically not bugs, or at least they are not filed as bugs. But under this new regime they are not allowed. So its off to Mantis to log a bug about the small change, then the commit.

I’m hoping after a while I’ll get into the pattern of logging bugs, then fixing them. Here’s hoping.

So after this review, I need to get some technicals into this document, as it will be my first reference when I forget how things actually work. In reality I installed SCMBug about 6 weeks ago, its only now that I had to make a change and forgot what it is that I realised I should have blogged it.

So here goes.

The installation of SCMBUG modifies the settings in the CVSROOT folder to setup pass through of the commits to the SCMBUG-server, this server needs to be started before any commits will actually be allowed.

Settings for how the SCMBUG server treats the commits are also in the CVSROOT directory, under the etc/scmbug directory. The file glue.conf should be checked out of CVS, edited and uploaded.

I think from memory the SCMBUG-server needs to be restarted for the changes to take effect, although to be honest I just edited the glue.conf file direct on the CVS server, not doing the checkout/in process.

In all it works well, when I commit I just need to add something like:

bug 22: Changed the way the login creates cookies
status 22: resolved fixed

The first line says add a log comment to bug 22, and record the changes.
The second line is to actually progress the workflow of the bug in Mantis, by resolving the bug 22 with a status of fixed.

Works well, any other updates will be posted to the site.