Transfer mode set to Awesome http://transfermodeawesome.posterous.com Dorkly bits on Exchange, Windows, software and whatever else I can think of. posterous.com Wed, 07 Oct 2009 21:21:00 -0700 Yet more Subversion repository polling headaches http://transfermodeawesome.posterous.com/2009/10/yet-more-subversion-repository-polling.html http://transfermodeawesome.posterous.com/2009/10/yet-more-subversion-repository-polling.html Either there is something very wrong with the way our repository is set up (i.e. my fault) or there is something wrong about the polling model for detecting changes in the repository.
This is the second or third time I’ve run into this issue.  Granted, we have not had the most fastidious design for our repository, however, I can’t believe that what we are doing on this small scale is so outside the pale of what other, much larger organizations (such as open-source hosters) are doing.  I just don’t think we’re that special.
The problem manifested itself today in the form of hundreds of authentications per second against Active Directory from a particular account and server.  The server is our Subversion repository server, naturally, and the account is the special build server account set up to monitor and build from our product locations in the repository.
The tide was sustained over a wide range of time and seemed to be contributing to visible load on the domain controller.  If not harmful, this is at the least embarrassing and annoying for our build infrastructure, which I’m very much invested in.  Nothing I know should be causing this kind of traffic, and if it is a consequence intrinsic to the system, I should find a way to make it less demanding.
I’ve run into similar issues with other tools before.  Evidently, polling a repository for changes is demanding on a repository.  We had an application called SVN monitor which was very nice.  Every time a commit was made to the repository, a notification would pop up from the system tray giving you the details, which you could then inspect with a pretty nice repository browser.  The polling interval could be configured as well.
Unfortunately, we would see enormous spikes in CPU usage on the SVN server when a few users would be polling for changes.  Limiting the number of polled locations and frequency of polling helped performance but greatly hampered the usefulness of the tool.  We eventually replaced it with the repository view from Redmine.
Back to tracking down the current issue.  The SVN server (VisualSVN) is configured for secure connections with Windows Authentication.  The authentication requests to AD coming from this server are responses to SVN clients making requests of the server.  Turning off the SVN server verified that it was indeed the source of the authentication requests.  There were only two servers configured to use that poll the subversion server using the account in question…Redmine and TeamCity, our build server.
The Redmine repository view does take some time (and presumably therefore, effort) to refresh its view.  It uses the subversion command line tools to do so.  However, it should only refresh when you visit the page.  So this isn’t a likely culprit unless something is very wrong.
TeamCity, on the other hand, is constantly polling to show you what changes might be pending for the next build.  This was the likely culprit.  Shutting it down quickly killed all of the authentication requests.  That was the winner.
However, what to do about it is an issue.  Obviously, it isn’t necessary to poll every 60 seconds for every VCS root in TeamCity, but there has to be a happy medium.  I reduced the polling interval globally to 10 minutes, which definitely helped.  There are only four VCS roots, and the TeamCity vcs log shows that they are polled individually each minute (by default)…so why the hundreds of authentication requests.  Something about the way TeamCity polls must break up the poll into lots of small requests, each requiring authentication.  This is what is nailing our domain controller.
Aside from reducing the polling interval, I’m not sure what else there is to do at the moment.  I haven’t posted a question on their support forum, but that’s on tap.  Perhaps there is more to come.

Permalink | Leave a comment  »

]]>
Fri, 14 Aug 2009 16:39:00 -0700 Redmine plugins http://transfermodeawesome.posterous.com/2009/08/redmine-plugins.html http://transfermodeawesome.posterous.com/2009/08/redmine-plugins.html I’ve found a number of the Redmine plugins to be innovative and look to be very useful.  Aside from the installation process being, well, command-line driven, they are simple to install and configure, for the most part.
Nothing against the venerable command-line.  I’m perfectly comfortable there.  However, with such a beautiful interface all around, seeing a prompt is fairly ugly in comparison.  And the actuality is that the platform-neutrality of a web interface gets muddied when you have to use the CLI due to the inevitable vagaries of cmd.exe versus bash, etc.  Proper environment variables and such are flies in the ointment.  Module and theme installation page in the Administration section, anyone?
But back to the plugins, the ones I found interesting were:
  • Code Review – automatically create and track issues by commenting on code in the repository.  Slick!
  • Question – Turn issue comments into questions directed at particular users (such as other developers who might have some useful knowledge).  Generates emails.  Great for knowledge capture that might otherwise be lost in forums or emails.
  • Embedded – Create a new tab that allows you to embed non-frames html files.  Useful for auto-generated html such as code documentation, test and coverage reports, etc.
Plugins that are interesting but I had trouble with:
  • Simple CI – Simple monitoring of continuous integration status via an RSS feed.  Looks rudimentary but useful, since the build server is the only important infrastructure not tied into Redmine, but it claimed my TeamCity RSS Feed was invalid even though it worked fine with Outlook.  More troubleshooting.
  • ezLibrarian – Resource reservation/tracking tab for things like books, machines, etc.  While it looks good in theory, the installation left incorrectly labeled tabs and fields.  Looks like an issue with the localization files (the author is not a native English-speaker).  The link given here is also not up to date, as I was able to get a more recent version from his github repository.  However, it did no better.  And what is Booty bay?
  • ezFAQ – from the same author.  A promising looking FAQ generator.  The docs say it doesn’t work with the 0.8.x series however, which is the latest stable.  Too bad.  It also modifies the Redmine core, which makes me leery.  Pass.
More to come on these as they actually see use in our production.
One note about configuring plugins after installation.  While most do not have a configuration section on the Administration > Plugins page, some do.  Some rely on custom fields in the projects, some do not.  Some add settings tabs in the projects.  All need to be enabled in Project > Settings > Modules.
The real trick, is however, that roles need to have permissions enabled on the new module as well.  Administration > Roles and Permissions takes you to the right place.  Look for the new module under each role and choose the appropriate permissions.  Most modules give no rights to any roles by default.

Permalink | Leave a comment  »

]]>
Fri, 14 Aug 2009 16:15:00 -0700 Relocating TeamCity build agents http://transfermodeawesome.posterous.com/2009/08/relocating-teamcity-build-agents.html http://transfermodeawesome.posterous.com/2009/08/relocating-teamcity-build-agents.html I logged into TeamCity today and was surprised to see no active agents.  Further investigation showed that I had entirely lost the connection between the server and the agents.  This is bad.
Fortunately, there was nothing seriously wrong.  We had moved the build agents and build server to a new VM server, which had necessitated converting the VM format and installing a new set of VM tools (VMware Server => ESX).  Evidently this was enough to confuse the TeamCity server.
After verifying that IP connectivity was working, I restarted build agents.  Upon restart, the agents reconnected to the server, however they were not recognized as the same as before.  This means I lost the configuration associated with these build agents.  Since our build agents are specialized to build for different environments, I lost the mappings for particular projects to go to particular build agents.
Fortunately, TeamCity makes this kind of stuff pretty easy.  A few clicks later and I was back in business.  While it would be nice if TeamCity were a bit more resilient to changes and could recognize the agents better, this wasn’t too difficult.

Permalink | Leave a comment  »

]]>