Posterous theme by Cory Watilo

Adding an OpenSearch provider for Redmine

Update: An OpenSearch plugin for Redmine has been created based on the work here.  I recommend you use that instead.  It is available at:

http://dev.holgerjust.de/projects/redmine-opensearch/wiki

Thanks to Holger for quickly turning this around.  Now, back to the original article:

Have you ever used your browser’s built-in search window?  Don’t answer that.

Internet Explorer and Firefox both allow search providers to be added to the built-in search window, so you can change which search the window actually performs.  By default my Firefox search comes with providers such as Google, Wikipedia, Answers.com, Amazon, etc.

I don’t actually use the feature much.  I’m used to typing Google searches in that box and don’t really think about anything else.  If I want to search Amazon, I go to Amazon.  But it is a convenience for Google.  The only way I might use it for something else were if there were another site I searched more often.

While it is arguable, I do search Redmine a lot for tickets, wiki info, etc.  As an experiment I wanted to switch my search box to Redmine.  I thought this would be a difficult task, perhaps involving coding.  Turns out is was much easier than I thought.  However, there was a lack of clear, simple steps to do so, so I’m sharing the process through which I went.

First of all, the search provider protocol, so to speak, is specified by OpenSearch.org.  I didn’t get a lot of help there.

I did, however, get help from a couple of sites if you just google “opensearch tutorial”.  While I still wasn’t sure exactly how to accomplish it, they made it a lot more comprehensible.

I also posted a discussion on the Redmine.org website, which landed me a key piece of help.  I thought I would have difficulty adapting the examples I was given, because I wanted to search an internal website not available on the web-at-large.  However, it wasn’t a problem after all.  I just couldn’t use the example on one web site that tried to use Google site search as the search provider.

The only caveat is that my implementation uses the most generic redmine search, which searches all projects.  More specific searches can be done after the first search.

There are two pieces you need:

  1. An opensearch description file available on your Redmine server
  2. A search provider notification link added to the template for Redmine’s html files, the one that includes the head tag

Here is the opensearch description file provided by another Redmine enthusiast, adapted for my site:

<?xml version="1.0" encoding="UTF-8"?>
<OpenSearchDescription xmlns="http://a9.com/-/spec/opensearch/1.1/">

    <ShortName>Redmine Search</ShortName>
    <Description>Search Redmine</Description>
    <Url template="http://redmine/search?q={searchTerms}"

         type="text/html" method="GET" />
    <Image height="16" width="16" type="image/vnd.microsoft.icon">http://redmine/favicon.ico</Image>
    <OutputEncoding>UTF-8</OutputEncoding>
    <InputEncoding>UTF-8</InputEncoding>
</OpenSearchDescription>

The reason for the edit above is that I previously posted another version of this file which turned out to be Firefox-specific, meaning it wouldn't work with IE.  This version works with both IE and Firefox.

As you can see, my server url is just http://redmine.  This won’t work for you, so you need to substitute your own.

You need to specify the template search string as well.  In the case of the general search, it’s simply your Redmine root url plus “/search”.  There are more specific project searches that include “/search/index/project-name”, where project-name is your project’s name, but that was too specific for my purposes.  You also want to tailor your search description.

Once that is customized, I needed to place it in the public directory on my Redmine server, which for me is c:\program files\bitnami redmine stack\apps\redmine\public.  Yours may be different.  I saved it as opensearchdescription.xml, which needs to be referenced elsewhere, so remember the name of yours.

Finally for this step, I wanted an icon associated with the search.  Unfortunately, Redmine hasn’t settled on a favicon as of my version, 0.8.4.  I couldn’t find a good one that is Redmine-specific, so I stole the one from ruby.org.  I also put this in the public directory, replacing the empty one there.

The other piece you need is a way to notify your browser that there is a search provider available.  This is done by a link present in the header of all of your pages on your site.  That requires modification of the base html template used by Redmine (other solutions are welcome, I don’t like customizing core Redmine files).

I added the following to apps/redmine/app/views/layout/base.rhtml, presented here in diff form:

<%= call_hook :view_layouts_base_html_head %>
<!-- page specific tags -->
<%= yield :header_tags -%>
+<link rel="search" href="http://redmine/opensearchdescription.xml"
+      type="application/opensearchdescription+xml"
+      title="Redmine" />
</head>
<body>
<div id="wrapper">

This is a link which I added right before the </head> tag.  It refers to the opensearchdescription.xml file I added earlier.

I also had to restart Redmine and clear my browser cache for everything to show up properly.

Once done, point your browser at your Redmine install and look for the “Add Redmine” entry in your search provider dropdown.  Once you add it, you still need to select the “Redmine” search provider entry.  This should do it.

Patching Redmine 0.8.x for email notifications on added news items regardless of user notification settings

That’s a mouthful.

Redmine 0.8.x does not send email notifications about added news items unless the user’s notification settings are set to permit updates about everything in the project, including all issue updates, etc.  This isn’t a very happy medium.

There is a ticket, #2549 at redmine.org that describes a “watch” feature for the news feed.  This would be ideal.  Since no real action seems to have been taken on it in the last year, I found a way to patch (read: hack) a dumb notification feature into the code.  I provide no warranty for this, it’s provided AS-IS.

The idea is that instead of sending to a project recipient list that has been filtered for users not wishing to see all notifications, to instead simply send to all project members.  To do this, I added a new method to the projects class which provides this unfiltered email list.  It requires two patches:

=== (+2,-2) apps/redmine/app/models/mailer.rb ===
@@ -101,7 +101,7 @@
   def news_added(news)
     redmine_headers 'Project' => news.project.identifier
-    recipients news.project.recipients
+    recipients news.project.news_recipients
     subject "[#{news.project.name}] #{l(:label_news)}: #{news.title}"
     body :news => news,
          :news_url => url_for(:controller => 'news', :action => 'show', :id => news)

and:

=== (+5,-1) apps/redmine/app/models/project.rb ===
@@ -205,6 +205,10 @@
     members.select {|m| m.mail_notification? || m.user.mail_notification?}.collect {|m| m.user.mail}
   end
+  def news_recipients
+    members.collect {|m| m.user.mail}
+  end

   # Returns an array of all custom fields enabled for project issues
   # (explictly associated custom fields and custom fields enabled for all projects)
   def all_issue_custom_fields

 

The second change create a new method, “news_recipients”, which is simply the email list of group members.  The first change makes the news_added event use the news_recipients list instead of the project’s regular recipients list.

This is a very naive approach, isn’t sanctioned by anyone, including me, and could easily have problems I don’t understand.  For example, I have no idea what this does with project members who have been locked out in the administration interface.  If security and privacy are important to your organization, do not use this patch.  You've been warned.

On a side note, I use Mercurial’s MQ extension to maintain such patches to source code on a production system.  This allows me to run with the patches until upgrade time, then to lift and reapply the patches after the upgrade process completes.

Interactive Python Console

One of the great things about Python is exploratory programming, the ability to “dry run” your logic and even design it while working within the interactive interpreter.

The usual Python console is fine for this, although some amenities are missing such as code completion suggestions.  Many more sophisticated options exist as well, such as the excellent IPython, which is basically an interactive console session on steroids.

Don’t get me wrong, IPython is fantastic, however, I’m a bit more GUI in my old age and I like having a variable list available in some corner window of my session.  I’ve always had a dissatisfaction with the GUI Python environments, however.  My former favorite was PyCrust, which is packaged along with the also-excellent wxPython.  It is nothing more than an interactive shell with a couple of those amenities like a variable list.  However, if you don’t do GUI programming much, wxPython is a heavy piece of software to install simply for the shell.

So I’ve tooled along with the interactive interpreter in Pydev, which is itself another piece of software I can heartily recommend.  As an IDE, while I am not entirely satisfied with the cobbled feel of it, Eclipse+Pydev is the best free option for a solid Python development environment that I’ve found.  Again, it’s pretty heavy for just a shell, but since that’s not what it’s primarily used for, you get the shell for free.

Unfortunately, Pydev’s interactive console functionality lags its other features, and it doesn’t have live watches in another pane of the IDE.  That’s ok, though, because there is a great alternative.

Whenever I need to do some Python development in a new environment, I don’t go through all the hassle of setting up Eclipse+Pydev.  Believe me, it takes a while.  There is a fantastic lightweight Python IDE that, as luck would have it, makes a great interactive console environment as well.  That’s PyScripter.  It’s developed in Delphi, meaning it’s fast and the GUI is very functional, while still having a small footprint and packing a lot of power.  If I didn’t like Pydev’s integration with source control and Pylint so much, I’d be using PyScripter (PyScripter also has Pylint integration, but it’s not automated).  While there are a lot of other good free Python IDE’s, and if were earning money with Python software development I might be willing to purchase ActiveState Komodo, PyScripter is a welcome balance of power and ease.

A note on setup.  PyScripter allows different layouts via drag-and-drop.  I put an interactive pane side-by-side with the editor pane by dragging the Python Interpreter tab up from the multi-tabbed pane.  You also want to use the Remote Python Interpreter configuration, which requires an older version of the Rpyc library.  That library is available in the downloads section of the PyScripter website.  Just unzip it and put it in Python’s lib\site-packages directory.  Finally, the wiki portion of the PyScripter website has themes you can download, if you don’t like the available UI looks.  I like the Vista Blue download.

Windows 7 Annoyances

In my experience upgrading to Windows 7, there are naturally some bumps in the road.  I have to say, it has been remarkably smooth and I am glad I skipped the entire Vista debacle (I replace Vista Basic with XP on my laptop when I got it years ago).

Here are a few things I ran into and what I had to do:

VMware VSphere

VMware VSphere will install, but it complains when trying to connect to the ESX server.  Unfortunately I didn’t record the error message (actually you get two separate ones), but anybody will experience it when using VSphere.  Fortunately, there is a solution:

http://www.techhead.co.uk/running-vmware-vsphere-client-on-windows-7

The only difficulty was editing the VpxClient.exe.config file.  For some reason, even though I had administrative access as well as file ownership, I couldn’t save my changes.  My IT administrator was able to save the changes for me, however.

The Essentials – Newbie edition

A friend of mine just recently bought a new machine and I wanted to point her to my post on the essentials, but I realized there’s probably way too much on the list and too little in the way of explanation.  Here’s my newbie edition of the list, a shorter version complete with explanatory text and links.

Before going through the list, I’ll point out a site that can save time in getting and installing many of these packages:

FreeApp allows you to select a number of these packages at once, then download a single installer.  That can save you a bunch of time.  Not all of these packages are available there, but many are.

On with the list.

Virus protection

Viruses are so prevalent today, it’s almost impossible to get away without virus protection.  While many computers come with virus protection, it is usually on a trial basis, after which they want your money.  There are good free anti-virus programs out there.  Microsoft’s new Security Essentials seems to be rated the best and is free.

That covers “real-time” protection, where every file you use is scanned for safety.  But sometimes viruses get past this.  When they do, the best scanner for removing viruses and malware that I’ve found is Malwarebytes.  You have to run the program from the start menu and tell it to scan your computer, so it’s not real-time protection, but it’s good at removing infections after the fact.

On-line backup

After virus protection, data protection is the next most important thing.  Even if your machine is protected from viruses, it isn’t protected from the ravages of old age, coffee spills or small children.

There are good on-line backup services that offer a modest amount of space for free, such as MozyHome Free.  If you refer others using your referral code, they even give you more space.  If you find this valuable, you can enter my referral code when asked by the configuration tool and I will receive some extra space for my backups.  Thanks in advance! My referral code is:

84MKKD

You sign up for a free subscription and install their software, which runs in the background and periodically backs up your important data.  It is smart and configurable, allowing you to specify the directories that store your data as well as the different file types you want backed up.  You can use it on up to two computers for the same account.

Web Browsing

  • Firefox

Friends don’t let friends use IE.  If you care about your computer, use Firefox.  Two essential add-ons, once you are using Firefox:

Download statusbar replaces the silly downloads window with a statusbar that is much easier to keep track of, and XMarks saves and synchronizes your bookmarks.  Not only that, it is excellent at suggesting good web sites that other people have bookmarked too.  XMarks requires you to sign up.

PDF Viewing

You can’t get away without looking at PDF files these days.  Unfortunately, Adobe Acrobat Reader has become bloatware.  Fortunately there are free, slimmer alternatives.  PDF-XChange Viewer is very functional and you do not have to purchase the Pro version.  This “trial” version works fine as free software without nagging you.

PDF Creation

Many times you may want to “print” a document to PDF instead of a real printer.  doPDF installs itself as a virtual printer, so that when you press the Print button in your favorite program, you can select the doPDF printer and have the printed version saved to a PDF file.  Works great.

CD Burning

CDBurnerXP is a free, good, simple solution for burning CDs.  I also like InfraRecorder because it is open-source, but I give a slight edge to CDBurnerXP for usability and versatility.

DVD and Video

While I do like Windows Media Player, it has never played DVDs for me on a fresh install.  I didn’t bother checking whether this is the case on Windows 7, I just loaded the solution I know works.

In this case, it’s not another application, it’s just a set of video “filters” that allow WMP to play DVDs, DivX and a host of other formats, including audio files that WMP isn’t capable of playing by default.  This is the best of both worlds…the WMP integration with Windows as well as the formats I want.  Just load these up and run Windows Media Player.

Photo Organizing and Touchup

Picasa is the bomb.  ‘Nuf said.

Saving Web Pages and Notes

I’m always getting receipts from on-line purchases and other sensitive web pages that I can’t just bookmark.  Rather than print them to PDF, where they got lost on my hard drive, I save them to Evernote, which is the best note-taking application I’ve ever seen.  Not only is it available on my computer, all of my notes are available over the web as well if I log into their site.  The service is free but requires a signup.

Searching for Files

Let’s face it, files get lost in the shuffle.  If you know the name of the file, or even part of the name, Everything will find it wherever it is on your hard drive, in a flash.  The limitation is that it doesn’t search inside the files, just the filenames, but that’s enough for me.  Solutions that go inside the files end up filling your hard drive with hundreds of megabytes, if not gigabytes of index information.  No thanks.

Cleaning and Organizing the Hard Drive

  • CCleaner
  • Disk Space Fan

When disk space starts to crunch, or when you just need to clean up the clutter, these two tools are indispensable.  CCleaner automatically finds and gets rid of unwanted Windows crap (that’s what the “C” is) that just builds up after a while.  Note: CCleaner will try to install Yahoo! toolbar whenever you install or upgrade it.  Remember to opt out of that, you don’t want it.

Disk Space Fan helps you find the rest.  It’s a simple application for finding out what files are taking up the most space on your hard drive.  Use CCleaner to clean up the stuff that belongs to Windows and use Disk Space Fan to clean up the stuff that belongs to you.

Deleted File Recovery

If a file isn’t in your recycle bin, then honestly I haven’t had much luck with deleted file recovery.  However, since it does happen, and it’s usually something important, I’ll mention the tool I’ve used in the past with mixed success.  I don’t think it’s the tool’s fault when things can’t be recovered, but it works less often than not in my experience.  Since I like the other tools by the same organization, though I keep recommending it.  You might look for other tools for the job as well, such as PhotoRec.

Again, there may be an option to install Yahoo! toolbar when installing or upgrading.  Opt out.

Honorable Mentions

These aren’t necessarily essentials, but I find them essential as a power user.

If you need to share files between computers, Windows Live Sync is excellent, and you don’t need to be a power user to find it useful, you just need to have more than one computer.

CubicExplorer is also a very nice file manager.

Defraggler comes with the same toolbar caveat that CCleaner has.

Enjoy.

The Essentials – Windows 7 edition

I recently converted my work machine to Windows 7, which is so far a good experience and one I’m happy with.  In the process, I gained a few new software tools and shed some old ones.  Here’s the list of what I consider the essentials for my environment on Windows 7 (not including those from Microsoft Update):

  • 7-zip
  • Adobe Flash Plugin
  • Beyond Compare 3 Pro
  • CCleaner
  • CDBurnerXP
  • CubicExplorer
  • Defraggler
  • DeskPins
  • Disk Space Fan
  • doPDF
  • Eclipse
  • Evernote
  • Everything (search)
  • ffdshow
  • HashCheck
  • Inkscape
  • Java
  • Launchy
  • LogMeIn
  • Malwarebytes Anti-malware
  • Microsoft Security Essentials
  • Mozilla Firefox
  • Mozilla Thunderbird (w/Lightning)
  • Notepad++
  • Paint.NET
  • PDF XChange Viewer
  • Picasa 3
  • Python 2.6
  • PyScripter
  • RDTabs
  • RENESIS Player Thumbnail plugin (SVG thumbnails in Explorer)
  • TortoiseHg
  • Virtual CloneDrive
  • Windows Live Sync
  • Windows Live Writer
  • XML Notepad 2007

If this were a home machine, MozyHome Free would also be on the list. I also have some tools, such as Visual Studio, installed on a separate machine, so those don’t make the list.

One interesting thing I learned during this experience is to not use the download links provided by Google when searching for a product.  More than one site had an old download link listed by Google, while visiting the main site search result and drilling to the downloads from there provided the most recent version.  Caveat emptor.

Another pleasant surprise was CubicExplorer.  While it lacks the dual-pane feature I like, it is otherwise a fine Windows Explorer substitute.  I have used several other programs, most recently UltraExplorer and before that xplorer2, but have always become frustrated with some lack of polish in this area.  CubicExplorer is simple and it works, while offering some nifty conveniences such as bookmarks that are simple to manage and, my new personal favorite, the ability to show the sort headers in any view style, details or not.  Nice.

Useful links for understanding Mercurial and Git

The basics

Comparisons

While the link below isn’t really a comparison so much as a set of suggestions for git, it happens to highlight some of the things I think Mercurial does right:

http://nubyonrails.com/articles/five-features-from-mercurial-that-would-make-git-suck-less

Unfortunately, almost all of the rest of the comparisons that show up on Google at the time of this writing are significantly out of date.  There are some good comparisons which enumerate what used to be valid reasons for choosing one system over the other, however, times are changing rapidly for these systems.  The following, while a bit out of date, are still relevant comparisons that have good highlights of the systems:

I list the Python PEP primarily because it contains some good workflow discussions.  The first two are a bit more up-to-date.

The one that reflects my own reasons for choosing Mercurial best is this:

http://www.dribin.org/dave/blog/archives/2007/12/30/why_mercurial/

I believe git repositories automatically optimize periodically now, so that’s less of a difference.  Mercurial now has hgsubversion to work with Subversion repositories as well.  Mercurial supports not only named branches within a repository but also git-style bookmarks, which differs from the article here too.

Finally, a pretty up-to-date and under-the-hood technical comparison of how they work is at Stack Overflow:

http://stackoverflow.com/questions/1598759/git-and-mercurial-compare-and-contrast

Git and Mercurial…mostly Mercurial

This is not so much a comparison of Git and Mercurial as a recounting of my experience looking at the two. If you are looking to make a decision between them, I can tell you two important things:

  1. All of my research indicates that when it comes to the technical capabilities of the two systems, they are so similar as to be almost indistinguishable. They both do the job and in almost exactly the same fashion. One will have a slight advantage over the other in any number of ways, but when it boils down, you won’t find anything compelling about one’s capabilities over the other.  And they are growing more similar over time as each implements the ideas of the other (at least, this is definitely the case with Mercurial borrowing features such as rebasing from git).
  2. That said, the environment in which you are operating makes a huge difference. The punchline is that, if you want to deploy a repository server and you run a windows-focused environment, then you have no choice but Mercurial (at the time of this writing).

In this article, I’ll cover:

  • Background and motivation for looking at distributed version control systems

In further articles I’ll cover:

  • Useful links for understanding and comparing the systems
  • Evaluating the capabilities of the systems
  • Working with Git on Windows
  • Experience with Mercurial

I’ll be writing a short series with more detail on my deeper experiences with Mercurial as well.

Motivation

The biggest motivation for looking at the so-called “next generation” of version control systems, the distributed version control systems or DVCSs, is that of the merge.

At my place of employment we currently use Subversion for version control purposes.  It’s a great system and one we’ve used for a long time, and I’ve personally used it even longer.  However, it works best when you don’t do a lot of branching.  Branches are treated as second-class citizens, a special case to be given special consideration only when actually attempting to merge results from the branch back into the trunk.  At that point, our experiences have indicated two things (even with the new merge tools introduced with Subversion 1.5):

  1. Merges rarely go off without a hitch.
  2. Merges make you go through a battery of options which require a thorough knowledge of the merge process.

The upshot of this is that it discourages use of branching.  Our head developer wants to worry about coding rather than the details of version control, and I don’t blame him.  A good rule for Subversion is to never let your branches drift far from your trunk, which requires frequent merging of trunk changes into your branches.  Because merges frequently encounter conflicts in Subversion (some new types of conflict were introduced with 1.5 as well which I still don’t understand and aren’t well documented), it’s just less of a headache to work on the trunk and leave branches out of the picture except for major feature-development.  Even then, it’s a pain.

Of course, working on just the trunk has its own issues, which version control should be suited to fixing, but can’t handle well without branching.  The biggest of these issues is that of parallel works-in-progress.  We are constantly working on issues with the code, resolving them (hopefully), then testing and iterating changes and fixes back into the code.  While these feedback cycles occur, the code is in a work-in-progress state, however, we can’t serialize the process.

Having our developers sit around twiddling their thumbs while the code is tested and vetted simply isn’t an option.  So they work on other issues while the feedback is generated from earlier work.  If you are only working on the trunk, this means there is always half-baked code in there.  When the time comes to cut further development and release, sometimes code isn’t fully baked but is too difficult to track and back out.  Thus bugs in the field are born.

So there is the appeal of distributed version control systems.  Branching allows you to track individual changes in their own branch and “graduate” them to the trunk after they have been vetted.  Not only that, if you want to test a fix in isolation from other trunk changes, the branch allows you to do this by implementing the fix on top of just the last released version of the code, without any interim work (vetted or not) to gum up the works.  While this kind of targeted development is sometimes more effort than it is worth (it means you have to track a lot more interim versions of the build and which particular problems they solve), it can be worthwhile a lot of the time.  If branches are low-cost (at merge-time as well as branch-time), then this model becomes possible.

That being the goal, let me say that that’s a theory for us.  Others whose posts I’ve read say the model works and the appeal is enough for us to investigate, but I personally haven’t seen this in practice yet.

In both Git and Mercurial (as well as the other contenders), branching is not only simple, it’s the only means of development.

Let me step back from that statement and explain what I mean.  In a DVCS, there is the concept of the repository and the working copy, just like in Subversion.  The repository contains the history of changes, and the working copy is the current state of the files, whether that is from the last revision in the repository or some other point, with or without your own local modifications.  The system tracks your changes to the working copy, allows you to go back to other points-in-time and generally handles change management on those files.

In Subversion, the repository is usually centralized somewhere on the network and is not only authoritative, it is the only repository for that code.  Sure, you can check in the same files somewhere else, but there is no system for keeping that repository in sync with the central repository, unless you look at distributed enhancements to Subversion, which I have not.

In a DVCS, there can be and always is more than one repository as soon as more than one developer is involved (and usually even before that).  The difference is that the working copies are almost inevitably tightly-coupled with a copy of the repository.  This means that the repository is local to your disk and resides alongside the working copy.  The upshot of this is that your working copy/repository is usually larger than the simple working copy of the centralized VCS, but in trade, repository operations are usually far faster.  More on the impact of this distinction later in my series.

The way this works is that there are mechanisms to keep your repository in sync with other repositories by pushing and/or pulling changes from one repository to another.  A typical workflow might be to have an “authoritative” or “build” repository that lives remotely, from which the developers copy theirs.  While the developers work, they can commit their changes to their local repository.  A commit does not automatically synchronize to other repositories.  Then, as a piece of work concludes, the work can be tested locally, approved, then pushed to the central repository from which other developers can then receive the changes by pulling.

Getting back to my point about branching being the regular mode of operation in this model, each repository in this case can be viewed as its own branch.  The reason I say this is because each copy of the repository is not required to synchronize after a change is committed.  In Subversion, once a commit is made, no other changes can be committed by other developers without them first updating their working copy and resolving any conflicts that arise.

This means that, in Subversion, for a given directory path there is only ever one line of commits (barring discussion about explicit branches, which are different directory paths).  Each subsequent commit contains all of the changes from every commit prior to it.

In a DVCS, concurrent changes can be committed to the various developers’ separate repositories, and no communication happens between them unless the developers initiate it manually.  This means that there are necessarily divergent lines of development which do not contain the same set of changes, even while each copy of the repository only sees its own straight line of development.

These unsynchronized copies of the repository are therefore each their own branch of development, which may continue unabated without having to be synchronized with the other branches.  In fact, nothing says that they need be synchronized ever again, although in practice they will usually be synchronized fairly frequently.

This kind of branching is called “anonymous branching” in Mercurial because each branch exists without any explicit name to mark it as such.  Two developers may separately develop branches that both think they are the “default” line of development.

At some point these changes will usually be synchronized.  When that happens, there is only one thing that the system can do…it must track two (or more) separate lines of changes that are not merged.  This is where the concept of “heads” comes in.  Each repository must store the branches of development which result in a different state of what is meant to be the same set of files at the ends of each branch.  Each branch then has its own head, the final state of files in that branch.

As a hypothetical aside, the system could force one of the developers to accept the others changes only into their working copy, resolve any conflicts, then update their single line of development with the merge.  By keeping the other changes out of the repository until they are merged, this could keep a single line of development in the repository, much like Subversion.  Concurrent development could continue without synchronization of this sort, but once synchronization occurred, there would be no branches.

However, this would be less flexible than what the DVCS systems actually allow.  By tracking separate branches with their own heads, synchronization can take place and separate lines of development can be shared between developers without requiring any sort of merge at synchronization time.  This means branched work can be shared between developers without requiring the work to be integrated.  Developers can collaboratively work on different development lines by synchronizing repositories and switching their working copy between the branched lines.

While there are other branch management tools in both systems, this fundamental form of branching is so intrinsic to git and Mercurial that they need to handle merging well by necessity because it happens frequently.

While this is a requirement for such systems, it is not an explanation of why merging is better/faster/easier/some combination than Subversions.  That’s a matter of implementation.  In practice, it is better.  Why?  I have no idea.  I haven’t played with it enough to know the differences and I don’t understand it well enough in Subversion in the first place to offer a rationale.  I’m sure that a large part of it is that these systems track file moves and renames explicitly and better than Subversion does.  Otherwise, I’m at a loss.  But it works, it really does.

Objectively, merging just works most of the time.  There is nothing to prevent you from introducing logical inconsistencies into your code by merging, and detection of actual file conflicts doesn’t prevent that.  However, in practice, that’s what testing (such as unit testing) is for, and most of the time the code works if your developers are working on logically separate portions of your code.  There are fewer merge options, which lowers the required knowledge and incidence of operator error.  Merges tend to go off without a hitch.

Of course, merges have to be done explicitly and then committed and pushed/pulled to other repositories, so there is a bit more elbow-grease to be applied in that respect, but it remains simpler than merging in Subversion by a great degree.

So, to recap, the motivation for us to look at these systems is to enable collaborative, multi-branch lines of development.  This will hopefully allow us to target fixes and features in a more fine-grained manner, with fewer interaction effects of work-in-progress.  It will also give us more control over released code, as branches must graduate to the main line of development.  We’ll see how these advantages pan out in practice as we get more comfortable with them.

Next time I’ll cover useful links for understanding and comparing git and Mercurial.

It’s been a while…

I’ve been lax in blogging for quite some time.  Time to get back on the horse.

Hmm.  Too much to go over everything since last time of course.   Some quick highlights of projects and directions, mostly work-related:

  • Git/Mercurial
  • Customizing Notepad++ and Eclipse appearance
  • Mockups/wireframes
  • Wiki knowledgebase technology
  • Integration with mojoPortal
  • MSBuild and Python
  • Processing XMind mindmaps

I’ll try to cover these in some detail in followup posts.  The blogs I enjoy best are the ones that cover some particular problem topic which I’m experiencing myself, to the point of detail that I can follow along and hopefully solve a problem.  I’ll try to make these posts useful in the same manner.

I could run a whole series on Git and Mercurial.  I guess I’ll start that now.

TortoiseSVN/Redmine integration

Spent the day working on integrating TortoiseSVN commit logs with Redmine issues.  Some cool stuff is possible.  This post is an adaptation of my status update for this task in my list, so it may not be completely coherent, but here it is:

Accomplished so far:

TortoiseSVN integration:
  1. Added bugtraq:url and bugtraq:logregex properties globally to repository folders
  2. Added webviewer:revision and webviewer:pathrevision properties globablly to repository folders
  3. Tested TortoiseSVN Redmine integration plugin

Redmine settings:

  1. Changed server repository settings so "fixed" keywords close and mark 100% done referenced issues

bugtraq folder properties

Configuration of bugtraq folder properties is described at http://tortoisesvn.net/docs/release/TortoiseSVN_en/tsvn-dug-bugtracker.html and http://stackoverflow.com/questions/516542/integrate-tortoisesvn-with-redmine.
These are properties which can be supported by any subversion client which chooses to implement them.  TortoiseSVN supports them.
The bugtraq:logregex property tells the client how to recognize a reference to an issue ID within the comment of a log.  It's value is a pair of regular expressions which tell the client how to recognize a reference to an issue ID and then how to extract the ID number from that phrase.
The pair of regular expressions used in this case is:
([Cc]loses|[Ii]ssue[Ii][Dd]|[Rr]eferences|[Rr]efs|[Ff]ixes(\s[Bb]ug)?)\s?#(\d+)
\d+
This allows the use of the following phrases to refer to an issue ID:
  • References: refs,references,IssueID
  • Fixes: fixes,closes,Fixes Bug
Using any of these phrases in a commit log message will cause TortoiseSVN to recognize it as an issue ID.  Tortoise will highlight the phrase in the commit to let you know it has been recognized.  Then, after committing, when you check the commit log that message will show up with a link for the ID number.  The bugtraq:url property determines how that id is turned into a link.  In this case, it points to the redmine server.  There is also a new column in the commit log which is just for issue ID so you can easily see issue ids in the log.
A log message may reference more than one issue id.  Each reference has to use the phrasing above.
The properties are set on folders.  In our case, they are set uniformly across all folders in the repository.  New folders do not get properties such as this created by default, but as long as most folders have them, it should still work.
It should still work because Tortoise examines the current working copy folder for the properties, and if not found, keeps looking higher in the working copy tree.  As long as one of the parent folders in the checkout has the properties, it will behave correctly.  Since they are currently set on every folder, the likelihood of the properties being found is high.
However, since checkouts are usually done at the project level (just for clarity: project not project group), then the properties need to be set at least at that level.  Therefore, any new project will need to have the properties set on it.  This can be done either explicitly on the project folder when it is created (preferably recursively after subfolders have been created as well).  Or it may be set recursively at a higher level in the directory tree.  This would need to be done either right after the project creation or perhaps by a periodic process sometime later.  A database commit hook could also enforce the setting of the property at the project level, but that would need development.

webviewer folder properties

The webviewer:revision property allows the TortoiseSVN client (and other clients) to relate the log back to a web-based view of the repository.  In this case, it relates to the repository view of Redmine.
When viewing the commit log for a file or folder in Tortoise, right-clicking the context menu when this property is set provides a "view revision in webviewer" context menu entry.  Clicking this entry takes you to the revision's changeset in redmine.  This is useful to see a more Redmine-integrated view of the repository than provided by TortoiseSVN.
The webviewer properties are set on folders similarly to the bugtraq properties and follows the same rules.
There is another property that allows you to go from the TortoiseSVN commit log to the Redmine view of a particular item (file or folder) at a particular revision.  That property is the webviewer:pathrevision property.  As mentioned above, it has been set.

TortoiseSVN Redmine integration plugin

TortoiseSVN issue tracker integration plugins are discussed under the heading Getting Information from the Issue Tracker at http://tortoisesvn.net/docs/release/TortoiseSVN_en/tsvn-dug-bugtracker.html.
TortoiseSVN integration plugins are modules that allow TortoiseSVN to retrieve an issue list from the issue tracker itself.  This makes it easier to remember the issue id associated with the changes being checked in.
Integration plugins are TortoiseSVN-specific and must be installed on the same machine as TortoiseSVN.  This means the plugin must be installed on each developer's machine.  It also means that other clients (such as the Visual Studio AnkhSVN plugin) do not get this capability as well.  This makes it easier to use Tortoise to do checkins, but is by no means a reason to abandon other clients.
The TortoiseSVN Redmine integration plugin can be obtained from http://code.google.com/p/chrismckee-project-bay/wiki/REDMINETSVN.
The Redmine plugin adds a button to the commit dialog in the upper right-hand corner.  The button is tied to the Redmine project URL by the configuration of the plugin.  Each Redmine project you are working on must have a configuration in the plugin in order for task lists to be available.
Because the plugin is sensitive to the project location of the working copy, it only shows those issues related to the project being checked in, which greatly simplifies things.  So the extra configuration is warranted and productive.
To configure the plugin, follow the directions at http://code.google.com/p/chrismckee-project-bay/wiki/REDMINETSVN#Setup.  The working copy directory is associated with the plugin and the project Atom feed from the issues list page for the project.  This configuration is done in the TortoiseSVN settings menu.  See the URL for full directions.

Redmine repository settings

The global settings for repositories in Redmine are at /redmine/settings">http://<your redmine server name>/redmine/settings under the Repositories tab.
By default, Redmine can respond to messages in the commit logs of checkins.  It automatically creates associations from an issue to a revision changeset when the issue is references in the commit log.  This makes it easier to keep track of the actual work of a task and changes made.
It can also take the action of closing an issue as well as setting the done percentage by responding to certain keywords in the commit log.
These features are similar to the earlier discussed TortoiseSVN capabilities associated with commit log keywords, but are implemented and configured separately.  The Redmine keyword recognition is performed separately from TortoiseSVN's.
There are two settings in the settings page referenced above.
The first is Referencing Keywords, which tell Redmine to make a link both in the issue as well as from the repository view.  The link in the issue refers to the repository change revision, and the link in the repository refers to the issue.
The second is Fixing keywords, which tell Redmine to set the status of the referenced issue as well as the Done percentage.
The current configuration recognizes the same words used in the TortoiseSVN integration.  refs, references, and IssueID make references.  fixes, closes, fixes bug change the status to Closed and set the Done percentage to 100%.  The ID itself must still be referenced by a pound-sign directly in front of the number such as #25.  Note that when an issue is closed, the link from the repository view still exists but has a strikethrough to indicate the closed status.