CA Agile Central's Git Connector Installation & User Guide

Note: This documentation applies to CA Agile Central's newest VCS-Git connector. If you are using Rally's original post-commit, hook-style Git connector, refer to the Git Legacy Installation & User Guide. Please note that the post-commit hook connector is deprecated and unsupported.

The Git to CA Agile Central connector helps you show traceability of code changes to artifacts in CA Agile Central.

Supported platforms

The connector supports running in a variety of platforms. This software is written in Ruby and has been tested and certified using Ruby 2.2.6. Subsequent to June 27, 2018, it will not work with any earlier version of Ruby. If this software is for use on a Linux/Unix/MacOSX system, then it must be installed after you install the httpclient and rally_api gems.

Prerequisites

  • Ruby 2.2.6  -  This is the supported version.

    The connector may not be compatible with Ruby 2.3.x and 2.4.x, we have not certified this connector with those versions, only 2.2.6 is certified. You may want to investigate how to use the Ruby version manager software (rvm) that makes installation and use of specific versions of Ruby simpler.

  • Ruby Development Kit for Ruby 2.0 to 2.3 (** Windows only **)
    1. Download the DevKit-mingw64-64-4.7.2-20130224-1432-sfx.exe from here.
    2. Add path to the devkit’s bin directory to environment Path variable.
    3. cd to the Ruby DevKit root directory.
    4. Run command: ruby dk.rb init
    5. Run command: ruby dk.rb install
  • Select a machine which will run the connector
  • Git must be installed on this machine
  • If the connector is on a different machine than the location of the Git server, note the server address and name
  • Know the paths to the repositories you will gather information from
  • Select the CA Agile Central workspace in which the information will be going

Download and install the connector

To download the connector, follow the steps here.

Installer and operator filesystem permissions

As this connector is intended to run on an ongoing basis initiated by either cron or Windows Task Scheduler (WTS), it is important to consider the identity of the user who installs the connector versus the identity of the user when running through cron/WTS with regard to the permissioning and ACL of the directories and files in the connector installation. The user running the connector must be able to read all files and must be able to write in the configs and logs subdirectories.

Distribution contents

1715 May 23 13:01 LICENSE
18419 May 23 13:01 README
4811 May 23 13:02 User-Mapping-Strategies
170 May 23 15:43 configs
136 May 23 13:02 extension
170 May 23 13:02 gems
1897 May 23 13:02 git2ca_agile_central.rb
2115 May 23 13:02 install_gems.rb
102 May 23 13:02 lib
136 May 23 15:36 logs
374 May 23 13:02 plugin
43520 May 23 13:02 vcseif-1.6.0.gem

Installation

  1. Unzip the git2ca_agile_central-(version)-master-(build_number).zip file into a suitable directory where you want the software installed.
  2. Set environment variable GEM_COMMAND to gem executable located in Ruby installation directory. The location examples on Linux and Windows respectively are:
    • /myhome/.rvm/rubies/ruby-2.2.6/bin/gem
    • C:\Ruby226\bin\gem
  3. Run the install_gems.rb Ruby script to get all associated gems installed. If you see output similar to You don't have write permissions ... then you will need to either consult with your system administrator to obtain the necessary permissions (typically done by using sudo or other similar software) or have the system administrator run the install_gems.rb command. To run the command, in a console/terminal window at the prompt, type the following:
    ruby install_gems.rb

Git server versus ssh

The primary use case is for this connector to be run on the platform that the Git software and repository reside on. While it is possible to run this connector on a platform that is not the Git platform, that use case assumes the use of ssh and the proper setup of public and private key information. Setting up ssh and key files is beyond the scope of this document; consult the internet for documents regarding ssh and PKI. If you are running the connector on Windows, you will need to have an executable for ssh if you plan to connect to a remote box. Packages such as Cygwin and the Git installer for Windows include an ssh.exe. You will then need to add the location of that ssh.exe to your PATH environment variable to be able successfully connect to a remote repository or server.

Set up Git

Consider how much Git history you want to have reflected in CA Agile Central

It is possible to operate the connector to catch-up with the complete history of commits in Git. You should assemble a pro and con list of considerations, reasons, and benefits to help you arrive at a strategy that works for your organization with respect to whether you need a complete commit history reflected in CA Agile Central or just reflected from some particular point in time.

Note: The Git connector will search the Git repository for commits. However, these commits get into the repository from the 'git push' command. The 'git commit' command is merely a local 'save' of your current work.

Lookback setting

The connector stores a timefile with the commit time of the last commit it successfully processed. Since Git is a distributed version control system, a certain number of days to lookback should be configured for the connector. This should be the best guess and comfort level for the way your team uses git. For example: Two team members are working on code in Git. One team member may push their commits up to the server every day. The second team member may only push their commits up at the end of the week. The connector will need to lookback to successfully find your second team member's commits. The Lookback setting in the configuration will lookback that number of days since the last successful commit to find commits to reflect in CA Agile Central. The default units are minutes, but you can use 5d, 5 Days, 24 hours, 24h as other values. As a best practice, we recommend a regular push strategy in your use of git to get those commits on the git server. The connector will check all refs in git for commit messages, so commits on a topic branch other than master will be found and reflected in CA Agile Central.

Set up CA Agile Central

Note: We strongly recommend creating the connector user with Workspace Administrator permissions. This will allow the connector user access to any new projects that may be added to the Workspace in the future (A user with Editor permissions only will not automatically be added to new projects). If the Connector User does not have access to projects, changesets will be created in the SCM Repository only and will be orphaned (not attached to any artifacts).

Verify that your target WorkspaceConfiguration has BuildandChangsetEnabled set. Your CA Agile Central workspace administrator needs to enable this for your target workspace. If it is not enabled, the connector will work, but you won't be able to see any of the changeset or change information that is associated with your CA Agile Central work items (story, defect , task).

Configure the connector

Copy the sample.yml file in configs to a new name, for example, repo_one.yaml. Edit the copied file and adjust some of the sample values for servers, credentials workspaces, repository names, and paths to values appropriate for your environment. The file is in YAML format. The YAML format uses indentation to represent structure and colon character (:) and dash character (-) to denote name and value separation and list elements. So, you'll need to be aware of preserving those syntactical elements in your edited file.

Within the Rally section of your YAML configuration file, there is an entry where you can name the SCMRepository in the Rally system to which changeset and change items will be associated. This SCMRepository is a named container which need not exist before running the connector; if it doesn't exist, the connector creates the SCMRepository in the Rally user's target workspace.

There is an optional lookback configuration item that can appear on either or both of the Rally and Git sections of the configuration file. The value for this parameter is expressed in terms of minutes (default or with the use of the m character) or hours using an h character or days by using a d character. Examples:

Lookback: 90
Lookback: 120 m
Lookback: 8 h
Lookback: 10 d

If you do not explicitly provide a lookback value, the connector uses a default value of 1 hour. CA Agile Central recommends that if you specify a value for lookback in either section that you also specify a lookback in the counterpart section with the same value. If the lookback values are not identical, there is the possibility under some circumstances that a changeset from Git would not be reflected in CA Agile Central due to the nature of distributed version control systems recording the original commit time and not the time the commits are pushed to a master repository; the window of time consideration being too short for the connector to pick up such commits.

Assess how user names are alike or differ in your Git system and in your CA Agile Central subscription . If the user names are universally identical in both systems, then you can either comment out all author sub-items underneath the Transforms section or you can set the value for the Author field in the Transforms section to Passthru. If there is a deterministic mapping that can transform a Git user value to the corresponding correct CA Agile Central user name value, you will need to adjust the transformation value for Author to the appropriate classname. Consult the User-Mapping-Strategies text document to determine which technique fits your particular situation.

Once you have identified a suitable Author transformation technique, you will need to edit the configuration to identify the Ruby class name that will implement that transformation. The Ruby class name must exist in a file of Ruby source code that is in a file in the plugin subdirectory.

Mapping SCMRepositories to VCS repositories

The VCS Connector was designed to operate where there is a one-to-one relationship between a Rally SCMRepository and a VCS repository (Git, GitHub, Subversion, Mercurial, and so on). A one-to-many mapping can create performance issues. When run, the VCS Connector creates a Rally SCMRepository item for the value specific in the connector config file if it does not already exist. This removes a small amount of administrative burden in that you do not have to create the Rally SCMRepository item before you run a configuration mentioning a new repository.

For each repository, there is a config file and a timefile. The timefile records the timestamp of the second-to-last commit for the repository. For example, you have an Apricot repository that had some flurry of commits last fall but nothing since and you also have a Banana repository that has had activity with the last couple of months. When commits for these two repositories get poured into a single Rally SCMRepository, it has the following effect. When processing the config for the Apricot repository, it is searching for recent Changeset records in Rally where recent is defined as the value in the timefile for Apricot. Since the last commit to that repository was last fall, there is going to be an excessive amount of information read-out of the single SCMRepository (because it is looking at all Changesets since last fall, not just the ones for Apricot).

Example configuration

 ---
VCSConnector:

Rally:
    Server              : us1.rallydev.com
    Protocol            : https
    Username            : [email protected]
    Password            : 22333four
#   APIKey              : _hgiotewhgeiwhgh325930503453490 # Used in place of Username/Password
    Workspace           : VanillaBean
    RepositoryName      : Balloon
#   Proxy               : some_proxy.yoursite.com:9876  # or an IP address:port
#   ProxyUser           : outbound
#   ProxyPassword       : bvc-357%GNQ
    Lookback            : 5 days  # in minutes by default, use m/h/d suffix for minutes/hours/days
    UpdateArtifactState : False
    StateExtractorClass : BasicActionsAndArtifactsExtractor(message)
    Debug               : False

Git:
#   Server         : your_vcs_server.company.com # optional, but if set connector runner user must authenticate using PKI
    RepositoryBase : /var/git/repo.git        #make sure the .git is specified for remote servers
#   RepositoryBase : C:\Users\Support\Git\GitRepo1        #Example of a Windows directory
    RevURI         : http://git.company.org/cgit/repo.git/commit/?id={revnumber}   # optional, if running web access to repo
    FileURI        : http://git.company.org/cgit/repo.git/tree/{filepath}/?id={revnumber}  # optional, if running web access
    Lookback       : 5 days
    Debug          : False

Services:
    Preview        : False
    LogLevel       : Debug

Transform:
    Author         : Passthru

...

RevURI and FileURI under Git config

This is meant to be the root URI if you are using a web front-end for your system. The URI will be used as a base for the link to the changesets and files. For example, if you have http://server:port/Git/rev/ as your root, a link will be made in Rally for the changeset to http://server:port/Git/rev/12345 for changset 12345. The strings {revnumber} and {filepath} are substituted in when the connector builds the link stored in Rally.

Note that the Proxy* items are commented out by using a # in front of the item. Within a YAML file, any line beginning with a # character is ignored and any content following a # character sequence is ignored (including the # sequence).

Create a Rally API key

A Rally API Key can be created to use to access your subscription data without using your username and password.

To use the API Key in a connector, edit the Rally section in the config .yml file and add an APIKey line. When the APIKey configuration is specified, omit the Username and Password from the Rally config section. If an APIKey entry is provided, the Username and Password are not used and a warning will appear in the log file if they are included in the config file. This is true even if the APIKey entry value is invalid, blank, or nil. If your subscription administrator has configured your subscription to be SSO only, you no longer need to add the user associated with the given APIKey value to the whitelist of authorized users.

The connector now uses rally_api version 1.2.1. Please note that the API Key must have full access, a read-only api key will not allow the connector to write to Rally.

For help creating a full-access API Key, please visit CA Agile Central Application Manager.

Run the connector

When developers put FormattedIDs, for example US42, in their commit messages, the connector processes all commits for configured repositories and pushes information about those commits into changeset objects and change objects in CA Agile Central. If a valid FormattedID is found in a commit message, the changeset created by the connector is associated with that defect, story, or task in CA Agile Central.

Example:

"Fixed DE1987 by changing preamble paragraph 3"

If Rally defect DE1987 had a state of Open prior to the commit and run of the connector, then subsequent to the operation of the connector processing this particular changeset, the state of Rally defect DE1987 would display as Fixed. Please note this message is case-sensitive, fixed is not the same as Fixed. The yml file will need UpdateArtifactState value set to True to process a state change.

The commit message may contain references to more than one artifact. For example, this is known to work:

"Test commit msg with multiple artifacts Fixed DE9 Closed DE8"

Within the Services section of your config file (repo_one.yaml for example), is an entry for Preview that is set to False. You may want to set this value to True in the initial setup phase to see that the connector can successfully connect to Rally with the credentials and information you provided, as well as proving out the use of the git command. Your PATH environment variable must contain a filesystem path where the git command can be found. See Linux / Unix / MacOSX / Windows documentation on how to set environment variables for use within a *nix cron job (or Windows Task Scheduler entry). Once you have determined that a connector run in Preview mode operates as expected, you can set the Preview value in your config to a False value.

On an ongoing basis, you can use cron or Windows Task Scheduler (or any other job or task scheduling software) to run the connector periodically. Initially, Rally recommends the connector to be run every 15 minutes during normal business hours and less frequently during non-business hours.

You can have numerous configuration files in the configs subdirectory and specify them all for invocation either by name or by globbing (wild- card syntax). For example:

ruby git2_agile_central.rb apricot banana cherry dogwood

The files apricot.yml, banana.yml, cherry.yml, and dogwood.yml must exist in the configs subdirectory. The connector only looks for config files in the configs subdirectory under the installation base directory.

Whenever the connector is run, an entry is made in the logs/git2rally.log file to note the invocation. For each config named at invocation, there is an entry in that file noting the return code from running that config. When the connector run has finished with all configs, an entry is written to note the completion of the connector run.

Additionally, there will be a file written in the logs subdirectory for each config named that will have much more detail about the activity that occurs during the run. You can adjust the verbosity written to these log files by modifying the LogLevel value in the config file. Normally, the LogLevel would be set to Info. Should you encounter situations where the connector does not complete normally, you can adjust the LogLevel to Debug and run the connector to begin the troubleshooting process. These log files can be sent to Rally Support to expedite the investigation of a case.

The connector will write a file in the base installation directory corresponding to the config name with the date of the last commit processed. The file is named config_time.file and has a time entry in the form YYYY-mm-dd HH:MM:SS Z (for Zulu time). When first run, there won't be a time file for the config and the connector defaults to looking for commits that occurred after 2010-01-01 00:00:00 Z. You can override that behavior by creating and editing a time file for the config you are about to process. By providing an entry in the format mentioned above, you can control that point from which commits are processed.

Note: The connector will skip processing commits for a repository until there are three or more commits to process. Once there are three or more commits, the "skipped" commits will be processed. This is a design decision due to performance issues in Agile Central when processing the commits.

Extensions

Within the extension sub-folder of the installation, there is an exemplar Ruby class in the file named statex.rb. The class definition contained in that file is BasicActionsAndArtifactsExtractor. This example class demonstrates the basic technique of examining a commit message and extracting Rally artifact identifiers and state transition target values. Using this class when the config file item UpdateArtifactState value is set to True results in the transition of the State (or ScheduleState in the case of UserStory) of the identified artifact to the state value mentioned in the commit message.

For example, if there is a Rally defect (identified as DE1987) mentioned in the commit message with a new valid state value either immediately preceding or following the artifact identifier, then the connector changes the state of the identified artifact in Rally to that state.

Example:

git  commit  my_file.java  -m  "Fixed DE1987 by changing preamble paragraph 3"

If Rally defect DE1987 had a state of Open prior to the commit and run of the connector, then subsequent to the operation of the connector processing this particular changeset, the state of Rally defect DE1987 would display as Fixed. Please note this message is case-sensitive, fixed is not the same as Fixed.

The commit message may contain references to more than one artifact. For example, this is known to work:

git  commit  my_file.java  -m  "Test commit msg with multiple artifacts Fixed DE9 Closed DE8"

The extension folder allows you to provide your own message processing to extract Rally artifact identifiers and state changes if the example provided does not fit your conventions. Your extension must be a class written in Ruby and must provide an instance method called service which takes the commit message as an argument and must return a Ruby Hash instance with entries keyed by a state name (Fixed, Completed, ...) or nil with a Ruby Array as the associated value populated by Rally artifact identifiers (FormattedID).

Troubleshooting

See log files for messages

When the connector runs, it generates two log files:

  • log/git2ca_agile_central.log—A global log file indicating the status of the connector.
  • logs/<config-file-name>.log—For each configuration file given on the command line, this will be the log file containing information about any issues with that configuration file.

The RepositoryName is case sensitive

The configuration file contains a line like RepositoryName: Chien1, which is the name of the SCMRepository object in Rally. This name is case-sensitive. Assuming the name was mistakenly specified as chien1, you may see a log file error like:

 INFO : RallyVCSConnection.validate - SCMRepository: chien1
DEBUG : RallyVCSConnection.getSCMRepository - No SCMRepository named |chien1| in Workspace: |WS1|
 INFO : RallyVCSConnection.createSCMRepository -
        Creating SCMRepository 'chien1' (Subversion) in the 'WS1' workspace...
ERROR : UnrecoverableException (/Users/...../exceptions.rb:42 line in `initialize') -
        Unable to create SCMRepository: 
        Error on request - https://rally1.rallydev.com/slm/webservice/1.40/scmrepository/create.js -
                {:errors=> ["Validation error: SCMRepository.Name:
                There is already a SCMRepository with the name 'chien1' in this Workspace"], ... ]}

Ruby Considerations

This software is written in Ruby and must be executed with a supported version of the Ruby interpreter. As of version 1.6.0 the supported Ruby version is 2.2.6. If you are running the connector on a Windows platform, you must have installed the appropriate Ruby DevKit. See Prerequisites for more information.

No Valid Artifacts Mentioned

Seen when connector tries to add a changeset to a Project where the Connector User does not have permissions.

DEBUG : RallyVCSConnection.extract_artifact_references - No valid artifacts were mentioned in message
DEBUG : RallyVCSConnection._createInternal - Created Changeset: OID: 4935999999  Revision: 14d0f533084c700f Timestamp:|2016-01-06T14:58:01.000Z| (not associated with any Artifacts)

Connector cannot connect when userid for proxy server has @ sign

Fixed by adding the addressable ruby gem.

[2018-01-30 06:58:39 Z]  INFO : RallyVCSConnection.connect - Proxy for connection to CA Agile Central: http://[email protected]:[email protected]:8080/
[2018-01-30 06:58:39 Z] DEBUG : RallyVCSConnection.rescue in connect - bad URI(is not URI?): http://[email protected]:[email protected]:8080/
[2018-01-30 06:58:39 Z] ERROR : OperationalError (/var/lib/gems/2.3.0/gems/vcseif-1.5.3/lib/vcseif/rally_vcs_connection.rb:334 line in `new') - Unable to connect to CA Agile Central at rally1.rallydev.com as user [email protected]

Issue: Error seen when Ruby gems are not installed.

Solution: Install ruby gems using install_gems.rb. Learn more.

/Users/test/.rvm/rubies/ruby-2.2.6/lib/ruby/site_ruby/2.2.0/rubygems/core_ext/kernel_require.rb:55:in `require': cannot load such file -- vcseif (LoadError)
    from /Users/test/.rvm/rubies/ruby-2.2.6/lib/ruby/site_ruby/2.2.0/rubygems/core_ext/kernel_require.rb:55:in `require'
    from git2ca_agile_central.rb:35:in `<main>'

Get help

If you have any questions, issues, or suggestions, click the Contact Support link at the bottom of any CA Agile Central page .

Revision history

  • 1.6.0-master --- 25-May-2018
    • Enhancements:
      • Unified distribution, requires use of Ruby 2.2.6 for TLS V1.2 support.
  • 1.5.3-master --- 12-Dec-2017
    • Enhancements:
      • Sync all vcs-eif connectors.
  • 1.5.1-master --- 12-Sept-2017
    • Enhancements:
      • Added Message to fields eligible for Transform.
      • Certified for use with Ruby 2.2.6.
  • 1.4.3-master --- 24-Oct-2016
    • Enhancements:
      • Updated the logic to allow for processing after 1 commit.
  • 1.4.1-master --- 17-Oct-2016
    • Fixes:
      • Addressed uncaught exception when processing an empty repository.
  • 1.4.0 - master-1046 (Nov-2015)
    • Enhancements:
      • Brand redesign.
  • 1.3.2 - master-1043 (Aug-2015)
    • Enhancements:
      • Added support for use of APIKey entry and revised documentation to encourage use of this
        in preference to use of still supported ApiKey.
  • 1.3.1 - master-1040 (Jul-2015)
    • Enhancements:
      • Added support for downcasing any Rally username value, inverted sequence of retrieval from Rally and target VCS system, and upped dependency on rally_api to 1.2.1.
  • 1.3.0 - master-1040 (April-2015)
    • Fixes:
      • Handle commit messages with multi-byte characters whose length exceeds the Rally Changeset Message field length limit of 4000 bytes.
    • Enhancements:
      • Updated dependency on rally_api to 1.2.0
  • 1.2.9 - master-1039 (March-2015)
    • Fixes:
      • Refined handling of commit messages in excess of 4000 characters to account for JSON encoding expansion.
  • 1.2.8 - master-1038 (March-2015)
    • Enhancements:
      • Improved the ability to identify artifact identifiers in commit messages surrounded by bracketing characters.
  • 1.2.6 - master-1036 (Jan-2015)
    • Enhancements:
      • Updated rally_vcs_connection.rb configureExtensionEnvironment and statex.rb to accommodate custom artifact prefixes (such as 'BUG' for Defect, and so on).
      • Updated dependency for rally_api to version 1.1.2.
  • 1.2.5 - master-1034 (October-2014)
    • Enhancements:
      • Updated to use Rally Web Services API v2.0 and the updated rally_api gem.
  • 1.2.4 - master-1032 (16-Sep-2014)
    • Enhancements/Fixes:
      • Added support for using API Key in lieu of username and password in the Rally portion of the configuration file.
      • Updated to rally_api toolkit version 1.0.1.
  • 1.2.3 - master-1030 (19-Aug-2014)
    • Enhancements/Fixes:
      • Fixed DE21993 - to meet Rally's commit message field limitation of 4000 characters, if a commit message exceeds the character limit, characters over the limit are truncated. Upon truncation, the commit message is appended with an annotation of the truncation and a warning message displays in the log file.

Feedback

Need more help? The CA Agile Central Community is your one-stop shop for self-service and support. To submit feedback or cases to CA Agile Central Support, find answers, and collaborate with others, please join us in the CA Agile Central Community.