CircleCI (v1) | Stitch Documentation

CircleCI is in beta
CircleCI is currently in beta. The info in this guide is subject to change.

CircleCI extraction is supported by Stitch
This integration is powered by Singer's CircleCI tap and certified by Stitch. Check out and contribute to the repo on GitHub.

For support, contact Support.

CircleCI integration summary

Stitch’s CircleCI integration replicates data using the CircleCI API v2. Refer to the Schema section for a list of objects available for replication.

CircleCI feature snapshot

A high-level look at Stitch's CircleCI (v1) integration, including release status, useful links, and the features supported in Stitch.

STITCH
Release status	Beta	Supported by	[Stitch] (https://community.qlik.com/t5/Support/ct-p/qlikSupport)
Stitch plan	Standard	API availability	Available
Singer GitHub repository	singer-io/tap-circle-ci
REPLICATION SETTINGS
Anchor Scheduling	Supported	Advanced Scheduling	Supported
Table-level reset	Unsupported	Configurable Replication Methods	Unsupported
DATA SELECTION
Table selection	Supported	Column selection	Supported
Select all	Supported
TRANSPARENCY
Extraction Logs	Supported	Loading Reports	Supported

Connecting CircleCI

Step 1: Create CircleCI API token

Log into your CircleCI application.
Go to your User settings.
Click Personal API Tokens.
Click the Create New Token button.
The the Token name field, type in a name you will remember - like Stitch Integration.
Click the Add API Token button.
Copy your new API token and paste it in a safe location, as you will not be able to view it in the application again.

Step 2: Create your project slugs

A project slug is a triplet of the componenets:

Project type - which can either be github or bitbucket
Organization - the name of your organization on GitHub or Bitbucket
Repository - the name of the repository

The project slug takes the form of <project_type>/<org_name>/<repo_name>. For example, the project slug for this CircleCI Singer tap is github/singer-io/circle-ci.

Determine which repositories you would like to include in your Stitch CircleCI integration and list them, separated by spaces, in the project slug format. Keep this list in a safe location.

Step 3: Add CircleCI as a Stitch data source

Sign into your Stitch account.
On the Stitch Dashboard page, click the Add Integration button.
Click the CircleCI icon.
Enter a name for the integration. This is the name that will display on the Stitch Dashboard for the integration; it’ll also be used to create the schema in your destination.

For example, the name “Stitch CircleCI” would create a schema called stitch_circleci in the destination. Note: Schema names cannot be changed after you save the integration.
In the Project Slugs field, paste the list of project slugs you created in step 2.
In the Token field, paste the API token you created in step 1.

Step 4: Define the historical replication start date

The Sync Historical Data setting defines the starting date for your CircleCI integration. This means that data equal to or newer than this date will be replicated to your data warehouse.

Change this setting if you want to replicate data beyond CircleCI’s default setting of 1 year. For a detailed look at historical replication jobs, check out the Syncing Historical SaaS Data guide.

Step 5: Create a replication schedule

Replication schedules affect the time Extraction begins, not the time to data loaded. Refer to the Replication Scheduling documentation for more information.

In the Replication Frequency section, you’ll create the integration’s replication schedule. An integration’s replication schedule determines how often Stitch runs a replication job, and the time that job begins.

CircleCI integrations support the following replication scheduling methods:

Replication Frequency
Anchor Scheduling
Advanced Scheduling using Cron (Advanced or Premium plans only)

To keep your row usage low, consider setting the integration to replicate less frequently. See the Understanding and Reducing Your Row Usage guide for tips on reducing your usage.

Step 6: Set objects to replicate

Is an object missing or not replicating? Verify that the object meets the requirements for selection and replication.

The last step is to select the tables and columns you want to replicate. Learn about the available tables for this integration.

Note: If a replication job is currently in progress, new selections won’t be used until the next job starts.

For CircleCI integrations, you can select:

Individual tables and columns
All tables and columns

Click the tabs to view instructions for each selection method.

In the integration’s Tables to Replicate tab, locate a table you want to replicate.
To track a table, click the checkbox next to the table’s name. A blue checkmark means the table is set to replicate.
To track a column, click the checkbox next to the column’s name. A blue checkmark means the column is set to replicate.
Repeat this process for all the tables and columns you want to replicate.
When finished, click the Finalize Your Selections button at the bottom of the screen to save your selections.

Important: Using the Select All feature will overwrite any previous selections. However, selections aren’t final until Finalize Your Selections is clicked. Clicking Cancel will restore your previous selections. Refer to the Select All guide for more info about this feature.

Click into the integration from the Stitch Dashboard page.
Click the Tables to Replicate tab.
In the list of tables, click the box next to the Table Names column.
In the menu that displays, click Track all Tables and Fields:
Click the Finalize Your Selections button at the bottom of the page to save your data selections.

Initial and historical replication jobs

After you finish setting up CircleCI, its Sync Status may show as Pending on either the Stitch Dashboard or in the Integration Details page.

For a new integration, a Pending status indicates that Stitch is in the process of scheduling the initial replication job for the integration. This may take some time to complete.

Initial replication jobs with Anchor Scheduling

If using Anchor Scheduling, an initial replication job may not kick off immediately. This depends on the selected Replication Frequency and Anchor Time. Refer to the Anchor Scheduling documentation for more information.

Free historical data loads

The first seven days of replication, beginning when data is first replicated, are free. Rows replicated from the new integration during this time won’t count towards your quota. Stitch offers this as a way of testing new integrations, measuring usage, and ensuring historical data volumes don’t quickly consume your quota.

Replication will continue after the seven days are over. If you’re no longer interested in this source, be sure to pause or delete the integration to prevent unwanted usage.

CircleCI table reference

Schemas and versioning

Schemas and naming conventions can change from version to version, so we recommend verifying your integration’s version before continuing.

The schema and info displayed below is for version 1 of this integration.

This is the latest version of the CircleCI integration.

Table and column names in your destination

Depending on your destination, table and column names may not appear as they are outlined below.

For example: Object names are lowercased in Redshift (CusTomERs > customers), while case is maintained in PostgreSQL destinations (CusTomERs > CusTomERs). Refer to the Loading Guide for your destination for more info.

jobs

The jobs table contains information about jobs from your CircleCI workflows.

Replication Method

Full Table

Primary Keys

_workflow_id

Useful links

jobs schema on GitHub

CircleCI API method

jobs table foreign keys

Join jobs with	on
pipelines	jobs._pipeline_id = pipelines.id
workflows	jobs._pipeline_id = workflows.pipeline_id jobs._workflow_id = workflows.id

Join jobs with

pipelines

jobs._pipeline_id = pipelines.id

workflows

jobs._pipeline_id = workflows.pipeline_id

jobs._workflow_id = workflows.id

jobs table schema

_pipeline_id

STRING

_workflow_id

STRING

approval_request_id

STRING

approved_by

STRING

canceled_by

STRING

dependencies

ARRAY

STRING

job_number

INTEGER

name

STRING

project_slug

STRING

started_at

DATE-TIME

status

STRING

stopped_at

DATE-TIME

type

STRING

pipelines

The pipelines table contains information about piplines from your CircleCI projects.

Replication Method	Key-based Incremental
Primary Key	id
Replication Key	updated_at
Useful links	pipelines schema on GitHub CircleCI API method

pipelines table foreign keys

Join pipelines with	on
jobs	pipelines.id = jobs._pipeline_id
workflows	pipelines.id = workflows.pipeline_id

pipelines table schema

created_at

DATE-TIME

errors

ARRAY

Click to expand errors

message

STRING

type

STRING

number

INTEGER

project_slug

STRING

state

STRING

trigger

OBJECT

Click to expand trigger

actor

OBJECT

Click to expand actor

avatar_url

STRING

login

STRING

received_at

DATE-TIME

type

STRING

trigger_parameters

OBJECT

Click to expand trigger_parameters

property_name

STRING

INTEGER

BOOLEAN

OBJECT

updated_at

DATE-TIME

vcs

OBJECT

Click to expand vcs

branch

STRING

commit

OBJECT

Click to expand commit

body

STRING

subject

STRING

origin_repository_url

STRING

provider_name

STRING

revision

STRING

tag

STRING

target_repository_url

STRING

workflows

The workflows table contains a list of workflows from your CircleCI projects, sorted by pipeline ID.

Replication Method	Key-based Incremental
Primary Key	id
Replication Key	created_at
Useful links	workflows schema on GitHub CircleCI API method

workflows table foreign keys

Join workflows with	on
pipelines	workflows.pipeline_id = pipelines.id
jobs	workflows.pipeline_id = jobs._pipeline_id workflows.id = jobs._workflow_id

Join workflows with

pipelines

workflows.pipeline_id = pipelines.id

jobs

workflows.pipeline_id = jobs._pipeline_id

workflows.id = jobs._workflow_id

workflows table schema

canceled_by

STRING

created_at

DATE-TIME

errored_by

STRING

name

STRING

pipeline_id

STRING

pipeline_number

INTEGER

project_slug

STRING

started_by

STRING

status

STRING

stopped_at

DATE-TIME

tag

STRING

Questions? Feedback?

Did this article help? If you have questions or feedback, feel free to submit a pull request with your suggestions, open an issue on GitHub, or reach out to us.

Related	Troubleshooting
Destination & Integration Compatibility Replication Scheduling Syncing Historical SaaS Data Resetting Replication Keys Nested Data Structures & Row Count Impact	Third-Party Downtime Understanding & Reducing Your Usage Re-Authorizing Integrations Replication Issues