The CA Agile Central VCS Connector for GitHub and GitHub Enterprise (GHE) posts information about GitHub repository commits to Agile Central (AC), and relates those commits to AC changesets and artifacts if sufficient information is contained in the VCS commit messages to identify the relevant artifacts.
"Fixed DE1987 by changing preamble paragraph 3"
If Agile Central defect DE1987 had a state of Open prior to the commit and run of the connector, then subsequent to the operation of the connector processing this particular changeset, the state of Agile Central defect DE1987 would display as Fixed. Please note this message is case-sensitive, fixed is not the same as Fixed. The .yml file will need UpdateArtifactState value set to True to process a state change.
The commit message may contain references to more than one artifact. For example, this is known to work:
"Test commit msg with multiple artifacts Fixed DE9 Closed DE8"
Pull requests are supported by the connector. To show pull requests in Agile Central as Third Party Context, add the PullRequests value set to True. The AC artifact formatted ID (ex. US1234) can be mentioned in either the pull request title or in one of the commit messages that is associated to the pull request in order to be processed by the connector. The connector will include open as well as a merged pull requests.
The Agile Central VCS Connector for GitHub is classified as a one-way and one-time mechanism. Information in GitHub is never altered; information is only written in Agile Central and no duplication of data is attempted or allowed. The Agile Central VCS Connector for GitHub consists of software that you run on your platform according to your desired schedule. The configuration of the connector is policy-based, meaning that you do not need to provide a separate configuration file for each GitHub repository for which you want the connector to operate. The policy-based nature allows you to scope to all or a subset of repositories accessible to a GitHub user or organization. The connector will only search master branches for commits.
If your organization uses GitHub Enterprise with SSO only authentication, set up the GitHub user as an Outside Collaborator. For more information on how to set up the GitHub user, see the following links:
This guide includes:
- Software Requirements
- Download the Connector
- Install the Connector
- Appendix A
The connector includes the following software requirements.
- A valid subscription and credentials to CA Agile Central.
- A computing platform running Linux or MacOS or Windows.
- An active account on GitHub (https://www.github.com).
- Supported on Python 3.5.x or 3.6.x (we recommend using the 64-bit version on Windows). You can retrieve the package installer from www.python.org.
- Windows-only additional requirements:
- The pywin32 package must be downloaded and installed subsequent to Python 3.5 or 3.6 installation.
- The pywin32 package is available from: https://sourceforge.net/projects/pywin32/files/pywin32/Build%20221/pywin32-221.win-amd64-py3.5.exe/download and https://sourceforge.net/projects/pywin32/files/pywin32/Build%20221/pywin32-221.win-amd64-py3.6.exe/download.
Download the Connector
To download the connector, follow the steps here.
You can install the connector several ways. There can be multiple install locations on the same machine that can run in parallel which determine the hardware required. We recommend the following guidelines:
- A computing platform on which your connector will run (Windows, Linux, Mac)
- Base hardware with a quad-core processor set and at least 8 GB RAM
- No more than 20–25 configuration files per install location
- Adequate disk storage capacity to store log files generated by the execution of the connector; we recommend 1 GB of free disk space
- If you add more install locations running in parallel, each installed location consumes up to 200 MB of RAM when running and less than 50 MB when idle
- 200–250 MB of space per install location is recommended since the required hard drive space is minimal and needs to scale only with log files
Install the Connector
After you download the connector, install it.
Follow these steps:
- Note: You must have Python 3.5.x or 3.6.x available in your environment.
- Unpack the ZIP file:
cd MyInstallDir # change your working directory to where you want to install the connector unzip vcsconn-2.0.3.zip # or use a suitable program to unzip the file cd vcsconn-2.0.3 ls -laR # observe the unpacked contents, or use dir on Windows # ghconn # ghconn module root directory # vcs2ac # connector initiation script, this is what you will run # configs # holds any config files used with this connector # sample.yml # a sample config to use as a base reference # sample_user_map.txt # a sample text file for user mapping # logs # hold log file and time file per configuration # README.txt # installation and users guide
**Go to the bottom of this document (Download Samples) for a sample file of a config with all the options available for this connector. The file is called: "Github sample file with explanatory notations.txt"
After you install the connector, set up both CA Agile Central and a configuration file.
Follow these steps:
- Set up CA Agile Central:
- Verify that your target WorkspaceConfiguration object has BuildandChangsetEnabled set to true.
- If not, your CA Agile Central workspace administrator needs to enable this for your target workspace. If it is not enabled, the connector will work, but you will not be able to see any of the changeset or change information that is associated with your Agile Central work items (story, defect, task).
- To enable this optiion, the workspace administrator must edit the workspace and do the following:
- Select the Setup icon .
- Select the Workspaces & Projects tab.
- On the Workspaces & Projects summary page, select your workspace.
- From the Actions drop-down, select Edit.
- On the workspace editor, select the checkbox labeled Enable Build and changeset.
- Select Save & Close.
- Set up a configuration file:
- Locate the config subdirectory.
- Copy the sample.yml file to a file named suitably for your environment (for example: "cp sample.yml to product_x.yml" --- or some other suitably named file that has a .yml suffix).
- Edit your product_x.yml file.
- Change the sample values for credentials, Workspace, Project, Job, View, Folder to values that are relevant and valid for your environment.
- See Appendix A on configuration file syntax.
Configuration of the github_connector is done through a YAML file which should be in the config subdirectory. There is a sample configuration file named sample.yml in the config subdirectory that you can use as a template. Copy it to a file with a name reflecting your intended usage or environment, and edit the file substituting with values relevant to your situation.
Follow these steps:
- Start simple.
- Locate the config subdirectory.
- Copy the sample.yml file to a file named suitably for your environment, such as copy sample.yml to product_x.yml.
- Edit your product_x.yml file. For example, change the sample values for credentials, workspace, project, token, and so on to values that are relevant and valid for your environment.
- Decide on the appropriate SecurityLevel.
- Decide whether you want to update state of Agile Central artifacts based on commit messages.
- Decide whether you want to map a GitHub committer to a valid Agile Central user and choose one of the three mapping methods: Passthrough, FileBasedUserNameLookup, or UserNameDomainAugmentLookup.
*See Appendix A on configuration file syntax.
You can configure operation to be either manual or scheduled.
Using a terminal window or console: cd to the installation root directory, for example /opt/local/sw/vcsconn-2.0.3. Then run:
python3 vcs2ac product_x.yml
This software requires that the configuration file reside in the config subdirectory. You specify the name of the file on the command line. Do not specify the subdirectory in the command line argument.
Use either cron, launchctl, or Windows Task Scheduler. Make sure the environment in effect when running this software has an appropriate environment set so that you can run:
python3 $VCS/github_connector your_config_file_name.yml
where $VCS is the reference to an environment variable containing the fully qualified path to the directory where the software is installed. For example, if you unzipped the package in /opt/local/sw, then your VCS would be set like this:
In normal operation, the connector writes a time file (in the log directory) whose name is based on the configuration file name. Example: If the configuration file name is product_x.yml, then the associated time file name would be product_x_time.file. The content of the time file is a single line containing a human readable date/time stamp value in the format YYYY-MM-DD hh:mm:ss Z. The value represents the timestamp of the last commit reflected in Agile Central. When the connector is run a subsequent time, it consults the time file to determine which jobs need to be considered for the current run by only processing the commits whose start time is greater than or equal than the time file value. It is possible to set the time file value artificially, (using ISO format, such as 2017-05-12 11:37:55 Z) so that you can go back in time and pick up commits from some arbitrary point in the past. The connector does not duplicate commits so you do not have to worry about duplicated information getting posted to Agile Central. In the absence of a time file, the connector pulls commits starting at a point three days prior to the current time.
For the purposes of reflecting VCS revisions/changesets in CA Agile Central, there is not always a direct one-to-one mapping between the value of the committer in the VCS and a CA Agile Central User. A CA Agile Central User is uniquely identified by the CA Agile Central UserName field value. The VCS connector has several methods available to enable the mapping of a VCS committer value to a CA Agile Central User. In the event that no mapping is found for a VCS committer identifier a Changeset will still be created in CA Agile Central, but there will be no value for the Author attribute in the Changeset. Consult the User-Mapping-Strategies text document (part of the connector installation) to determine which technique fits your needs.
The following information will be needed in the config file in the VCSConnector section (just after "Project"). Please consult the README_*.txt file in the main installation directory for all options.
Transforms : # R #Author : Passthrough # This is the default setting if not provided Author : FileBasedUserNameLookup(user_map.txt) #Author : UserNameDomainAugmentLookup(chiselor.com)
To support another SaaS VCS, you will need to develop a connection spoke for your build system that can support a call to getOrganizations, and getRecentRepositoryActivity(ref_timestamp). That method needs to return a list of commit items, where a commit item has the following attributes:
The connector always writes a log file named based on the configuration file name. The log file is written into the logs subdirectory under the base installation directory. You can control the extent of logging by specifying the level in the configuration file. Within the configuration file, you can set the LogLevel which determines the amount of logging information written to the log file. Under the Service section, modify the LogLevel setting (valid values are ERROR, WARN, INFO, DEBUG). When you set the LogLevel to DEBUG, you will get the full range of logging messages that can be helpful in pinpointing where a problem occurred and what information was being processed at the time of the anomaly.
It can be very helpful to run the connector in Preview mode when setting things up for the first time. This allows you to get the connections to Agile Central and GitHub to initialize and validate correctly without posting any changesets.
*Issue: Pull Requests are not transferring to Agile Central.
Answer: Verify the following line is in the config file:
PullRequests : True # Default is False, when True VSTS PullRequests are detected and reflected in Agile Central
*Issue: Error seen when using * in the include statement to pull all repos.
FATAL: VCSConnectorRunner.run - VCSConnectorRunner.getConfiguration vcs_connector_runner.py(288) - Unable to parse config/vsts-cloud.yml successfully, while scanning an alias in "<unicode string>", line 25, column 26: Include : * ^ expected alphabetic or numeric character, but found '\n' in "<unicode string>", line 25, column 27: Include : * ^
Answer: Escape the * in the include statement with quotes.
Repository: Include : "*"
When credentials in the configuration file are no longer in clear text, SecurityLevel settings can only upgraded, but not downgraded. For example, a configuration file with encrypted credentials cannot be used with SecurityLevel reset from Encrypted to Encoded. If users want to downgrade security level, they have to start with a configuration file where credentials are in clear text.
Appendix A - Configuration File Editing
The Agile Central github_connector for GitHub uses a text file in the YAML format. For complete information, consult the web page at or any other website on the subject of YAML. Please see the Download Samples section at the bottom of this page for a sample configuration file with explanatory notes.
For brevity, this document mentions several of the most relevant syntax items and covers the three sections of a valid YAML configuration that can be used with the connector.
- Use a text editor (not MS Word or Google Doc) to edit the file.
- Never use a tab character in the file. YAML does not allow or recognize tab characters.
- Save the file in UTF-8 format.
- Use a monospace font.
- Be consistent with the number of spaces you use to indent.
- On a line, the first occurrence of a non-quoted # character indicates a comment, the # char and all chars to the right are ignored in processing.
- Keep the sections in the same order as is present in the sample.yml file.
- Be aware that the colon character : is significant, it separates a key from the value.
- Be aware that the dash character - is significant, it denotes the start of a list which may have one or more key value pairs that constitute the list item.
- You usually do not have to quote values if they have spaces in them; you will have to quote the value if it contains an embedded # character.
Skeleton of the template_config.yml file
GitHub: AgileCentral: ... # several key value pairs are relevant for this section GitHub: ... # several key value pairs are relevant for this section Service: ... # a few key value pairs relevant for the overall operation of the connector appear in this section
The AgileCentral section specifies values to use to obtain a connection with Agile Central. The GitHub section specifies values to use to obtain a connection with GitHub and specify the policies governing what repositories in GitHub get processed to result in posting of changesets to Agile Central. You must specify the server for your GitHub Enterprise in the GitHub section of the configuration file. The Service section controls some aspects of the connector behavior on an overall basis. For example, For example, Pull Requests can be added to this section as True.
- 2.0.3-Master --- 9-Feb-2018
- Fixed name of StateExtractor class in sample config file
- Fixed assignment of Changeset URI for Bitbucket Server
- 2.0.0-Master --- 21-Dec-2017
- Unified all newer policy based VCS connectors into one distribution package
- 1.3.2-Master --- 7-Dec-2017
- Support for appending committer name to commit message
- 1.3.0-Master --- 28-Nov-2017
- Added support for pull requests
- 1.2.2-Master --- 16-Oct-2017
- Included pull request title to be parsed for formatted id
- 1.1.1-Master --- 12-Sept-2017
- Added support for GitHub Enterprise (GHE) as of version 2.10.0
- 1.0.0-master --- 30-May-2017
- Initial release