VCS Connector for VSTS 2017/TFS 2017

Overview

The CA Agile Central VCS Connector for VSTS 2017/TFS 2017 posts information about VSTS 2017 and TFS 2017 commits to CA Agile Central, relating those commits to CA Agile Central changesets and artifacts if sufficient information is contained in the VCS commit messages to identify the relevant artifacts. This connector works with TFS 2017 when the TFS server is configured to use git for version control. The connector will also update the state of relevant CA Agile Central artifacts, specifically defect state or user story schedule state when commit messages contain appropriate syntax, for example US123 In-Progress. The CA Agile Central VCS Connector for VSTS 2017/TFS 2017 is classified as a one-way and one-time mechanism. Information in VSTS 2017 and TFS 2017 is never altered; information is only written in Agile Central and no duplication of data is attempted or allowed.

The Agile Central VCS Connector includes functionality that detects the creation of VSTS and TFS pull request items and posts corresponding pull request items in Agile Central when the configuration option for this is present in the target configuration file.

The Agile Central VCS Connector for VSTS 2017/TFS 2017 consists of software that you run on your platform according to your desired schedule. The configuration of the connector is policy-based, meaning that you do not need to provide a separate configuration file for each VSTS or TFS repository for which you want the connector to operate. The policy-based nature allows you to scope to all or a subset of repositories within a VSTS project (or accessible to a TFS user in a collection).

This guide includes:

Software Requirements

Connector Download

To download the connector, follow the steps here.

Hardware

You can install the connector several ways. There can be multiple install locations on the same machine that can run in parallel which determine the hardware required. We recommend the following guidelines:

  • A computing platform on which your connector will run (Windows, Linux, Mac)
  • Base hardware with a quad-core processor set and at least 8 GB RAM
  • No more than 20–25 configuration files per install location
  • Adequate disk storage capacity to store log files generated by the execution of the connector; we recommend 1 GB of free disk space
  • If you add more install locations running in parallel, each installed location consumes up to 200 MB of RAM when running and less than 50 MB when idle
  • 200–250 MB of space per install location is recommended since the required hard drive space is minimal and needs to scale only with log files

Installation

 

**Please navigate to the bottom of this document to the "Downloads samples" section for a sample file of a VSTS or TFS config with all the options available for this connector.

 

  1. You must have Python 3.5.x or 3.6.x available in your environment.
  2. Run these commands: 
    pip3 install requests==2.18.4
    pip3 install pyral==1.4.0
    pip3 install PyYAML==3.12
    pip3 install cryptography==1.7.1
  3. Unpack vcsconn-2.0.0.zip. Change your working directory (cd) to a directory where you want to install the connector. Unzip vcsconn-2.0.0.zip (or use a suitable program that can unzip a .zip file): 
    cd vcsconn-2.0.0.zip
    ls -laR # observe the unpacked contents, or use dir on Windows
    
       vcsconn                     # vcsconn module root directory
       vcs2ac                      # connector initiation script, this is what you will run
       config                      # holds any configuration files used with this connector
          sample_vsts.yml          # a sample configuration to use as a base reference 
          sample_user_map.yml      # a sample text file for user mapping
       log                         # hold log file and time file per configuration
       README_VSTS.txt             # this file
       User-Mapping-Strategies.txt # details VSTS/TFS 2017 user to AC username mapping strategies

Setup

General recommendations:

  1. Start simple.

  2. Identify the person most familiar with the setup and administration of your VSTS 2017/TFS 2017 and Agile Central subscriptions.
  3. Locate the configuration subdirectory.

  4. Copy the sample_vsts.yml or sample_tfsgit.yml file to a file named suitably for your environment for example, cp sample_vsts.yml to product_x.yml (or some other suitably named file that has a .yml suffix).

  5. Edit your product_x.yml file. For example, change the sample values for credentials, workspace , project, token, and so on to values that are relevant and valid for your environment.

  6. Decide on the appropriate SecurityLevel.

  7. Decide if you want to update state of CA Agile Central artifacts based on commit messages.

  8. Decide if you want to map a VSTS or TFS committer to a valid Agile Central user and choose one of the three mapping methods: Passthrough, FileBasedUserNameLookup, or UserNameDomainAugmentLookup.

  9. Decide whether you want to process repositories for a specific user or for a named organization.

 

       *See Appendix A on configuration file syntax.

Operation

Manual

Using a terminal window or console: cd to the installation root directory, for example /opt/local/sw/vcsconn-2.0.0. Then run python3 vcs2ac product_x.yml. This software requires that the configuration file reside in the configuration subdirectory. You specify the name of the file on the command line. Do not specify the subdirectory in the command line argument.

Scheduled

Use either cron, launchctl, or Windows Task Scheduler. Make sure the environment in effect when running this software has an appropriate environment set so that you can run: python3 $VCS/vcs2ac your_config_file_name.yml where $VCS is the reference to an environment variable containing the fully qualified path to the directory where the software is installed. For example, if you unzipped the package in /opt/local/sw, then your VCS would be set like this: export VCS=/opt/local/sw/vcsconn-2.0.0.

Updates

If you re-install or update the connector, save a copy of your configuration files so that you do not have to redo your configuration work.

Time File

In normal operation, the connector writes a time file (in the log directory) whose name is based on the configuration file name. Example: If the configuration file name is product_x.yml, then the associated time file name would be product_x_time.file. The content of the time file is a single line containing a human readable date/time stamp value in the format YYYY-MM-DD hh:mm:ss Z. The value represents the timestamp of the last commit reflected in Agile Central. When the connector is run a subsequent time, it consults the time file to determine which jobs need to be considered for the current run by only processing the commits whose start time is greater than or equal than the time file value. It is possible to set the time file value artificially (using ISO format, such as 2017-05-12 11:37:55 Z), so that you can go back in time and pick up commits from some arbitrary point in the past. The connector does not duplicate commits so you do not have to worry about duplicated information getting posted to Agile Central. In the absence of a time file, the connector pulls commits starting at a point three days prior to the current time.

Troubleshooting

The connector always writes a log file named based on the configuration file name. The log file is written into the log subdirectory under the base installation directory. You can control the extent of logging by specifying the level in the configuration file. Within the configuration file, you can set the LogLevel which determines the amount of logging information written to the log file. Under the Service section, modify the LogLevel setting (valid values are ERROR, WARN, INFO, DEBUG). When you set the LogLevel to DEBUG, you will get the full range of logging messages that can be helpful in pinpointing where a problem occurred and what information was being processed at the time of the anomaly.

It can be very helpful to run the connector in Preview mode when setting things up for the first time. This allows you to get the connections to Agile Central and VSTS/TFS to initialize and validate correctly without posting any changesets.

Issues and Errors

 

*Issue: Pull Requests are not transferring to Agile Central.
Answer: Verify the following line is in the config file:

PullRequests : True # Default is False, when True VSTS PullRequests are detected and reflected in Agile Central 

*Issue: Error seen when using * in the include statement to pull all repos. 

FATAL: VCSConnectorRunner.run - VCSConnectorRunner.getConfiguration vcs_connector_runner.py(288) - Unable to parse config/vsts-cloud.yml successfully, while scanning an alias
  in "<unicode string>", line 25, column 26:
                Include   :  *
                             ^
expected alphabetic or numeric character, but found '\n'
  in "<unicode string>", line 25, column 27:
                Include   :  *
                              ^

Answer:  Escape the * in the include statement with quotes.

 Repository:
            Include   : "*"

 

*Issue: Server error seen in the logfile in VSTS connector version 2.4.0 

ERROR: VCSConnectorRunner.run - 'Server'

 


Answer: This issue will be addressed in the next release . The temporary workaround for this issue is adding the following line to the config file in the AgileCentral Section: 

Server : rally1.rallydev.com

 

 

Security Level

When credentials in the configuration file are no longer in clear text, SecurityLevel settings can only upgraded, but not downgraded. For example, a configuration file with encrypted credentials cannot be used with SecurityLevel reset from Encrypted to Encoded. If users want to downgrade security level, they have to start with a configuration file where credentials are in clear text.

Appendix A - Configuration file editing

The Agile Central vcs2ac connector for VSTS/TFS uses a text file in the YAML format. For complete information, consult the web page at or any other website on the subject of YAML. Please see the Download Samples section at the bottom of this page for a sample configuration file with explanatory notes.

For brevity, this document mentions several of the most relevant syntax items and covers the three sections of a valid YAML configuration that can be used with the connector.

  • Use a text editor (not MS Word or Google Doc) to edit the file.
  • Never use a tab character in the file. YAML does not allow or recognize tab characters.
  • Save the file in UTF-8 format.
  • Use a monospace font.
  • Be consistent with the number of spaces you use to indent.
  • On a line, the first occurrence of a non-quoted # character indicates a comment, the # character and all characters to the right are ignored in processing.
  • Keep the sections in the same order as is present in the sample.yml file.
  • Be aware that the colon character : is significant, it separates a key from the value.
  • Be aware that the dash character - is significant, it denotes the start of a list which may have one or more key value pairs that constitute the list item.
  • You usually do not have to quote values if they have spaces in them; you will have to quote the value if it contains an embedded # character.


Skeleton of the template_config.yml file 

 VCSConnector:
         AgileCentral:
             ...  # several key value pairs are relevant for this section
         VSTS (or TFSGit):
             ...  # several key value pairs are relevant for this section
         Service:
             ...  # a few key value pairs relevant for the overall operation 
                  # of the connector appear in this section

The AgileCentral section specifies values to use to obtain a connection with Agile Central. The VSTS or TFSGit section specifies values to use to obtain a connection with VSTS/TFS and specifies the policies governing what repositories in VSTS/TFS get processed to result in posting of changesets to Agile Central. The Service section controls some aspects of the connector behavior on an overall basis. Please consult the README_VSTS.txt or README_TFSGIT.txt file in the main installation directory of the connector for all options that can be used with this connector.

Revision History

  • 2.0.0-Master --- 21-Dec-2017
    • Initial release supporting VSTS and TFS 2017 (for projects using Git)

Feedback

Need more help? The CA Agile Central Community is your one-stop shop for self-service and support. To submit feedback or cases to CA Agile Central Support, find answers, and collaborate with others, please join us in the CA Agile Central Community.