Alfresco on EC2

Over the weekend, I created a Alfresco Labs 3b AMI on EC2, Amazon's cloud computing platform.

I took one of the Alestic Ubuntu 8.10 base images, added my own ec2-tools_0.1.deb package, and built out an AMI with Labs 3b running on the system tomcat5.5, instead of the bundled tomcat instance. That part was far more brutal than using EC2. You have to make quiet a few changes to the catalina policy to get things working.

I made an Alfresco package, that installs an /etc/tomcat5.5/policy.d/60alfresco.policy file that looks like this:

grant { 
    permission java.lang.RuntimePermission "accessClassInPackage.org.apache.*";

    permission java.lang.RuntimePermission "accessDeclaredMembers";    
    permission java.lang.reflect.ReflectPermission "suppressAccessChecks";
    permission java.util.PropertyPermission "alfresco.jmx.dir", "read,write";
    permission java.util.PropertyPermission "webapp.root", "read,write";
    permission java.io.FilePermission "/usr/share/java/servlet-api-2.4.jar", "read";
};

grant codeBase "file:${catalina.home}/bin/tomcat-juli.jar" {
    permission java.io.FilePermission "/usr/share/tomcat5.5/webapps/alfresco/WEB-INF/classes/logging.properties", "read";
    permission java.io.FilePermission "/var/lib/tomcat5.5/temp/-", "read,write,delete,execute";
    permission java.io.FilePermission "/var/lib/tomcat5.5/temp", "read,write,execute";
}

All of my AMIs have a rebundle.sh script that can quickly upload an updated AMI. It looks something like this:

#!/bin/sh
ACCOUNTID=xxxxxx
CERTFILE=/etc/ec2/xxxxxxx.pem
KEYFILE=/etc/ec2/xxxxxxx.pem
ACCESSKEY=xxxxxxxxxxx
SECRETKEY=xxxxxxxxxx

umount /var/local
ec2-bundle-vol -u $ACCOUNTID -c $CERTFILE -k $KEYFILE -p ubuntu-8.10-appsuite-1.0-20081101 --ec2cert /etc/ec2/amitools/cert-ec2.pem -r i386 
ec2-upload-bundle -b nvizn.com -m /tmp/ubuntu-8.10-appsuite-1.0-20081101.manifest.xml -a $ACCESSKEY -s $SECRETKEY
ec2-register nvizn.com/ubuntu-8.10-appsuite-1.0-20081101.manifest.xml

This made life a bit easier as I made changes to the image and uploaded them. I unmount /var/local at the start of the script as that's where I mount my EBS volume.

Samba4 on Ubuntu Intrepid

Here's a brief rundown of my experiences with Samba4 on Ubuntu Intrepid.

I first tried the samba4 package in the ubuntu intrepid repositories, but when you do a

./setup/provision --realm=azulogic.com --domain=azulogic --adminpass=fubar --server-role='domain controller'

you get a python stackdump with

IOError: [Errno 2] No such file or directory: '/usr/etc/samba/smb.conf'

I tried creating a "/usr/etc/samba" folder (though the distaste was high), but then proceeded to get further file path errors.

So, next I switched to the Debian Experimental package. This worked much better.

After you apt-get install the package, you'll have to fixup /etc/init.d/samba4 - it's still looking for smbd (the samba3 daemon), whereas in samba4 its now /usr/sbin/samba.

So, I just did a

ln -s /usr/sbin/samba /usr/sbin/smbd

to get it to work.

After getting krb5, dns, and samba ready to go, I tried to join a linux machine running winbind 2:3.2.3-1ubuntu3 to the domain. No luck though:

(~) net ads join -U Administrator
Enter Administrator's password:
Failed to join domain: failed to lookup DC info for domain 'AZULOGIC.COM' over rpc: NT_STATUS_INTERNAL_ERROR

How do you fix this? One way is to run in the "single" process model mode. I changed /etc/init.d/samba4 to launch the samba daemon with -M single. Then you see a nice:

(~) net ads join -U Administrator
Enter Administrator's password:
Using short domain name -- AZULOGIC
Joined 'LTS' to realm 'azulogic.com

One final note: as far as I can tell the debian version (4.0.0alpha6-GIT-7fb9007) crashes when someone tries to do a change password. So beware!

Secure Apt Repository Howto

After a good bit of googling and poking around, I completed the setup of our secure apt repository here at nvizn.

Here's how you'd do it for an Ubuntu intrepid repository.

First, setup a directory tree that looks like this:

mkdir -p /var/www/packages/dists/intrepid/main/binary-i386/
mkdir -p /var/www/packages/intrepid/main

Then, install apt-ftparchive, which will do most of the heavy lifting.

apt-get install apt-ftparchive

Now, drop all your .debs into /var/www/packages/intrepid/main/ and create an apt-ftparchive configuration file at /etc/archive.config

Here's what mine looks like:

Dir {
  ArchiveDir "/var/www/packages";
  CacheDir "/home/joel.reed/uploads/";
};

Default {
  Packages::Compress ". gzip bzip2";
  Sources::Compress ". gzip bzip2";
  Contents::Compress ". gzip bzip2";
};

APT::FTPArchive::Release::Codename "intrepid";
APT::FTPArchive::Release::Suite "intrepid";
APT::FTPArchive::Release::Origin "Joel W. Reed";

TreeDefault {
  BinCacheDB "packages-$(SECTION)-$(ARCH).db";
  Directory "intrepid/$(SECTION)";
  Packages "$(DIST)/$(SECTION)/binary-$(ARCH)/Packages";
  SrcDirectory "intrepid/$(SECTION)";
  Sources "$(DIST)/$(SECTION)/source/Sources";
  Contents "$(DIST)/Contents-$(ARCH)";
};

Tree "dists/intrepid" {
   Sections "main";
   Architectures "i386";
}

Finally, run this sequence of commands:

apt-ftparchive generate /etc/archive.config
cd /var/www/packages/dists/intrepid/
apt-ftparchive -c /etc/archive.config release . > Release
rm -v Release.gpg
gpg -v --output Release.gpg -ba Release

When you're done, you'll end up with a /var/www/packages tree that looks something like this:

/var/www/packages/dists/intrepid
/var/www/packages/dists/intrepid/main
/var/www/packages/dists/intrepid/main/binary-i386
/var/www/packages/dists/intrepid/main/binary-i386/Packages.gz
/var/www/packages/dists/intrepid/main/binary-i386/Packages.bz2
/var/www/packages/dists/intrepid/main/binary-i386/Packages
/var/www/packages/dists/intrepid/Contents-i386
/var/www/packages/dists/intrepid/Release
/var/www/packages/dists/intrepid/Release.gpg
/var/www/packages/dists/intrepid/Contents-i386.gz
/var/www/packages/dists/intrepid/Contents-i386.bz2
/var/www/packages/intrepid
/var/www/packages/intrepid/main
/var/www/packages/intrepid/main/alfresco-r3184-0.3.1.deb
/var/www/packages/intrepid/main/nvizn-base-0.3.6.deb
/var/www/packages/intrepid/main/libnss-cache_0.1-1_i386.deb
/var/www/packages/intrepid/main/nsscache_0.8.4.1_all.deb
/var/www/packages/intrepid/main/stratus-desktop-0.2.deb
/var/www/packages/intrepid/main/packages-main-i386.db
/var/www/packages/intrepid/main/jsetup_0.5.1_all.deb

Now, to make all this work, you need to have a gpg key of course, and apache set to serve up /var/www/packages, and all client machines need the public key. To do that with a key on a keyserver, do something like

gpg --recv-keys B1850655 && gpg --export B1850655 | apt-key add -

Hope this is helpful to you!

Startup

I haven't blogged for while, because I've been putting a lot of hours into an open source startup company. It's been great fun to work with some new technologies like Groovy, Grails, CouchDB, and Samba4.

Among other things, I setup an openldap server, built a few custom www.openldap.org/lists/openldap-software/200807/msg00002.html">overlays, and integrated Zimbra, Alfresco, Openfire, SipX, Samba3, and an Ubuntu desktop. Each of these integrations has there pros and cons, perhaps Zimbra and SipX are the nicest.

I'm hoping to blog about my experience with Samba4 shortly.

OpenTF 0.6.0 Release

Wow - two months without a blog post and 3 months since my last OpenTF release! For the last month or two, I really haven't worked much on OpenTF, preferring instead to work on learning NT Greek and more about the Book of Isaiah.

The latest release includes a few new goodies and many bugfixes. There's the new IRC changeset notification bot, support for CruiseControl (an open source continuous build framework), a monodevelop plugin for browsing TFS servers, and several new commands like "shelve", "rollback", and "merges".

Over the next few month, I hope to be able to further develop the monodevelop plugin, continue work on missing commands, and begin testing other open source Team Foundation tools for compatibility with the OpenTF libraries.

Job Openings

If there's anyone looking for a ASP.Net developer position in the Pittsburgh (PA) area and you've contributed to the mono project in the past, please put a link to your resume in the comments for this post. We're a great company to work and are early adopters of .Net related technologies. I'd love to be able to hire folks who have helped out the mono project. Thanks!

MonoDevelop and Team Foundation

Now that MonoDevelop is nearing a 1.0 release, I thought I'd take another look at fleshing out a TeamFoundation plugin for MD.

For starters, I'm taking the "tf explore" command in OpenTF, factoring out the Gtk classes into a separate assembly, then building out an MD addin that makes use of it.

Obligatory screenshot:

It will take a while to clean up this code and the build machinery, and to figure out how to make better use of builtin MonoDevelop addin services, but perhaps in a release of two, we'll have something useful. If you're interested in helping, please do!

At some point, I'm also hopeful that those developing the VersionControl API for MD can consider the needs of a Team Foundation plugin. I'd be very interested in seeing if we can make something work for SVN, GIT, etc. and TFS. That'd be a much better situation.

OpenTF Build Changes

I started the OpenTF project out by copying the Mono Olive tree, and replacing its assemblies and tools with my Team Foundation files. This worked well on *nix, but recently I've been trying to improve support for building on Windows as well.

Should I use cscript, nmake, powershell, BAT, project files, or some combination of these? How could I implement a build solution that didn't just duplicate the same build instructions (source files, references, etc) in 2 different formats: one for windows and one for *nix?

I decided to keep things simple on Windows - just use a VS2005 solution with a bunch of project files. Then for *nix, I decided to make libxslt's xsltproc a build requirement, and generate the list of sources and references for the mono olive make machinery using a few simple XSL stylesheets.

For example, all the .sources files are now generated via build/sources.xsl. Which looks something like this:

  <xsl:template match="/">
    <xsl:apply-templates select="Project/ItemGroup/Compile"/>
  </xsl:template>

  <xsl:template match="Compile">
    <xsl:value-of select="@Include" /><xsl:text> </xsl:text>
  </xsl:template>

I also have .references files for each assembly, also generated via an XSL file from the .csproj.

Now, I just maintain the VS2005 project files, and leave the *nix build stuff to the stylesheets. I added support for conditional sources using the Conditional attribute. Its working quite well thus far.

Using git-svn with Mono

Why use git to hack on mono?

I've found myself far more productive and make heavy use of feature branches and squashed commits for my day job, so when I hack on mono, I really enjoy being able to leverage the same capabilities.

By "squashed commits", I guess I should really say, leveraging the power of a distributed version control system that lets me break down a task into many smaller steps, commit each step individually, then squashing the whole thing down to one patch that I can post to mono-devel for review.

So do you set things up to use git with Mono?

cd /usr/local/src/
mkdir mono && cd mono
mkdir mcs && cd mcs
git-svn init svn+ssh://username@mono-cvs.ximian.com/source/trunk/mcs
git-svn fetch -r 86200 && git-svn fetch
cd ..
mkdir mono && cd mono
git-svn init svn+ssh://username@mono-cvs.ximian.com/source/trunk/mono
git-svn fetch -r 86200 && git-svn fetch

Note: the above recipe copies the svn history only back to revision 86200. You can pick any valid svn revision number you like, or if you want the full revision history see this page on the Mono wiki.

Ok. Everything's setup. Now what?

First, let's say I want to hack on some ASP.NET ashx page bug. I'll setup a local branch "ashx" to store whatever code I write/change:

git-checkout -b ashx

Now I have two branches: "master" which was setup by git-svn above, and "ashx" which I just created and switched over to. Now, I can:

emacs -nw class/System.Web/...
git-commit -a -m "1st step"
emacs -nw class/System.Web/...
git-commit -a -m "2nd step"
emacs -nw class/System.Web/...
git-commit -a -m "3rd step"

Ok, now to post a message to mono-devel:

git-diff master ashx > ~/Bug6884.fix
mutt

When everything looks good and no one has any complaints, I can finally commit back to mono's svn repository with:

git-branch master
git-pull --squash --summary . ashx
git-commit -a -m "message for mono's svn"
git-svn dcommit

This "squashes" my commits down into one batch of changes on the master branch, which I then commit and push to svn repo.

Finally, How do I update my local tree?

git-svn fetch && git-svn rebase remotes/git-svn

Note: This will update the current branch you are on locally.

Hope someone finds this helpful!

test RPM for tf4mono

I just caught up on my reading of the mono mailing list and saw Miguel's post about Mono Packaged .NET apps for Mono.

Since I have debian packages and a win32 installer for tf4mono, I thought it might be time to make an RPM package as well and maybe help this QA effort.

Anyway, I downloaded the very helpful Mono 1.2.5 VMWare image and went to work on creating a spec file for rpmbuild. Side note: cleaning out the bash history and ~/.ssh might be a sensible improvement to this image.

I had to try and remember all the old rpm command line options I used to use in my sleep - as I fell in love with debian's apt-get several years ago and forgot most rpm incantations.

Anyway, here's the resultant RPM package. By the way, I enabled the optional gtksourceview-sharp based syntax highlighting in the package.

If anyone can review the tfs.spec.in file or the RPM file and offer suggestions for improvement, please do so. I'll gladly make a necessary cleanups.

I installed the package and ran "tf show build" and "tf show stats /server:my.tfs.server.ip" and "tf explore /server:my.tfs.server.ip" and everything seemed in order. The "tf show" commands are new in the soon to be released 0.5.2 version of tf4mono.

Monthly Sleep Deprivation

Oddly enough, I've seem to have fallen into a schedule of releasing updates
to tf4mono about once a month. This month's release is
tf4mono 0.5.1 which includes win32 installation packages, a GTK-based gui mode
for exploring TFS repositories, many command enhancements, improved builtin help with
usage guidelines, and numerous bugfixes.

I always need a bit of downtime after a release first of course -
working on open source software as a hobby is fun,
but always ends up meaning lost sleep every so often.

Anyway, I'm interested in hearing what features would make tf4mono
more useful to you. Better support for locking files? Handling merge conflicts?
Easier building on win32 platforms? More GUI support? Let me know!

The master/trunk branch of tf4mono just got a "stats" command which makes use of
/VersionControl/v1.0/administration.asmx to generate some server statistics.

Here's some sample output:

(~/Source/tfs-lsg-1.0) tf stats
Files:           812421
Folders:         20033
Groups:          481
Pending Changes: 7907
Shelvesets:      180
Users:           184                                                                         
Workspaces:      154

I plan on augmenting this output a bit, but its a good start.

The most nagging issue for me is actually a NTLM bug in mono that I keep hoping someone will eventually fix. It could be bug
#80687, though I'm not sure of it. On windows boxen, tf4mono never gives occasional auth failures, but on mono it does - especially on a fast network.

By accident, I noticed that I never saw any auth failures when working from home
over a VPN, but at work I'd see the auth failures quite regularly. If I route my
TFS traffic at work thru my home machine, thru the VPN, and back to work I never see auth failures. So it seems the faster the network the more likely you are to see this NTLM bug in mono.

tf4mono for windows

Thanks to Nullsoft Scriptable Install System, I've created some win32 installation packages for tf4mono.

There are two install options.

The first, tf4mono-base-0.5.1-rc1.exe, has been compiled without any GUI code. It has no external dependencies and should run on any win32 box with the .Net 2.0 framework installed.

The second package, tf4mono-full-0.5.1-rc1.exe, includes the graphical TF explore command. To run this version on win32, you must first install the Gtk# Installer for Windows.

Neither package adds the tf4mono installation folder to the SYSTEM or USER path. You'll have to do this by hand for now.

If you're on a windows box and have a few minutes to test out the package, please do so. Any feedback on how they work would be awesome.

Explore Command

Rather than try to learn how to write a monodevelop plugin and the Gtk# API all at the same time, I thought I'd start out by just getting my head around Gtk#.

Consequently, I've been working on a tf explore command that popups a Gtk# Application that lets you explore the TFS repository you're connected to. Here's what it looks like thus far:

I've been pretty impressed with the available widgets and their ease of use.

I can't quite figure out yet how to show a WATCH cursor when the TF query is lengthy - I've tried several different incantations but all to no avail.

That's the only thing that seems like it should be easier.

Read-Write Local Trees in TFS

One rather annoying feature of TFS is that all files are marked read-only in your local tree until you checkout a file, at which point it becomes read-write. Its not hard to find other people who also find this very annoying.

I was hopeful MS would make read-write local trees an option in the Orcas release of TFS, but recently came across this post by Buck Hodges where he explains it won't be a feature in Orcas.

I was annoyed enough to add support for read-write files in version 0.5.0 of tf4mono.
To enable it just type

tf config File.ReadWrite true

Then from that point on tf get, tf undo, tf checkin, tf add and friends will do the right thing.

If you know what you're doing you can compile and run the tf4mono client on Windows, either against the MS TFS assemblies or with the tf4mono implementation of these assemblies.

I didn't want to lose this flexibility but to really do this right API level support would be required. Subsequently, I ended up teaching the tf4mono TFS assemblies to read the TF client XML config file. This is slightly distasteful but the standard assemblies already read the workspace cache XML file so I can live with it. I really didn't want to bolt this read-write stuff on top of the existing code, because that would have meant alot more stat's and chmod's under the hood.

Anyway to really make this work I also had to beef up the tf online command, but that will have to wait for another blog post.

Mono.GetOptions working overtime

When I started tf4mono, the command line Team Foundation client, I didn't really want to invent yet another option parsing library and it seemed like Mono.GetOptions would fit the bill.

Over time though I found myself adding more and more options to my Options.cs file, which really got out of hand. Finally, I found myself in a hotel room in Tulsa, Oklahoma with no internet access - a perfect time for code cleanups!

I ended up keeping global options in my Options.cs and moving all command specific options into each *Command.cs file, then I give both the global driver and the desired command a crack at parsing the arg array. With a little extra hackery to support command chaining, I ended up being able to do a huge code cleanup and really enhance the builtin tf help command.

While I was mucking around in there, I added a custom "Command" attribute to decorate my Command classes. Now each command in the tf client looks something like this:

[Command("history", "Display changelog history for specified file.")]
class HistoryCommand : Command
{
[Option("Recursive", "R", "recursive")]
private bool OptionRecursive = false;

[Option("Stop After", "", "stopafter")]
private int OptionStopAfter = -1;

With a little reflection, the tf client now has a reasonable tf help <cmd> feature, which gives you:

(/usr/local/src/tfs/tools/tf) tf help hist
history (alias hist)
Display changelog history for specified file.

Valid options:
/format:ARG        Format "brief" or "detailed" (also /F:ARG)
/recursive         Recursive (also /R)
/stopafter:ARG     Stop After
/workspace:ARG     Workspace name (also /W:ARG)

Now I just need to figure out a way to generate the asciidoc manpage source file from the builtin help and life will be golden.