Jira connector tutorial part I

In this article, we'll review a use case for the Elastic Jira native connector. We'll use a mock project where a bank is developing a money transfer app and needs to integrate the information in Jira into Elastic.

The native connector allows us to get into our Elastic cluster information from tickets, tasks, and other documents, centralizing data and enabling advanced search features.

The main benefits of using this connector are:

Data from Jira is synchronized with Elasticsearch.
Access to advanced search features.
Document Level security (DLs) matching source security. You can only search for what you're allowed to see in Jira.

steps

Configuring Jira connector

You'll first need to get an API token from Jira to connect to Elasticsearch. Go to this link to learn how to create it.

Name it "elastic-connector." It should look like this:

Get the token and to your Kibana dashboard. Then, go to native connectors and select New Jira Cloud connector.

https://<YOUR_KIBANA_URL>/app/enterprise_search/content/connectors/new_connector?service_type=jira&connector_type=native

Replace YOUR_KIBANA_URL with Kibana endpoint.

Name the connector “bank” and click “Create and attach an index named bank” to create a new index with the same name.

Done! Now we need to configure our Jira data.

We'll keep "Enable ssL" off since we won't be using our own ssL certificates.

You can see the details of each field in the official documentation.

Activate Document Level security (DLs) so you get your documents with the users and groups authorized to see them.

Once the connector is correctly configured, you can continue to synchronize data as you can see below. It might take a couple of minutes to get the data from Jira.

Full Content: indexes all Jira documents.
Incremental Content: only indexes changes from the last Full Content sync.
Access Control: indexes Jira users in the security index to activate DLs.

We can check the connector's Overview to see if the sync was successful.

In the Documents tab, we can see exactly what data we got with the connector. The objects from this first sync are:

Projects
Issues
Attachments

Indexing documents into Elasticsearch

We are not limited to searching across the connector documents. Elasticsearch allows you to search on many indices with a single query.

For our example, we'll index additional documents into the galactic_documents index to see how search works with more than one datasource:

Compliance Manual of the GBFF
User Guide for the Galactic Banking App
Technical specifications Report

But before indexing, we'll create optimized mappings for each field:

PUT /galactic_documents
{
  "mappings": {
    "properties": {
      "document_id": {
        "type": "keyword"
      },
      "title": {
        "type": "text",
        "fields": {
          "raw": {
            "type": "keyword"
          }
        }
      },
      "content": {
        "type": "text"
      },
      "release_date": {
        "type": "date",
        "format": "yyyy-MM-dd"
      },
      "page_count": {
        "type": "integer"
      },
      "tags": {
        "type": "keyword"
      }
    }
  }
}

With the mappings configured, we can now index:

POsT galactic_documents/_bulk
{ "index": { "_index": "galactic_documents", "_id": "1" } }
{ "document_id": "GBFF-001", "title": "Compliance Manual of the GBFF", "content": "This document sets forth the compliance standards for intergalactic financial entities: Quantum-level data encryption to guarantee security in all transactions. Mandatory multi-factor authentication for all users and administrators. Quarterly reviews of security policies and access audit logs.", "release_date": "2024-01-01", "page_count": 5, "tags": ["compliance", "security"] }
{ "index": { "_index": "galactic_documents", "_id": "2" } }
{ "document_id": "GBFF-002", "title": "User Guide for the Galactic Banking App", "content": "Welcome to the Galactic Banking application by Interstellar Finance Corp. Here you can: Transfer galactic credits to any registered account across the Milky Way. Check your balance and manage your investments in real-time. Access interplanetary loans with ease. For your security, use multi-factor authentication each time you log in.", "release_date": "2024-01-01", "page_count": 3, "tags": ["user guide", "application"] }
{ "index": { "_index": "galactic_documents", "_id": "3" } }
{ "document_id": "GBFF-003", "title": "Technical specifications Report - Galactic Banking Project", "content": "This report details the technical architecture of the Galactic Banking application: Microservices-based backend for scalability and performance. secure communication protocols utilizing quantum encryption. Transaction management adapted to environments with gravity variations and time dilation.", "release_date": "2024-01-01", "page_count": 7, "tags": ["technical", "specifications", "architecture"] }

Querying data

Now that we have both Jira objects and documents, we can search for them together.

GET bank,galactic_documents/_search
{
  "query": {
    "multi_match": {
      "query": "galactic moon",
      "fields": [
        "content",
        "title",
        "*description",
        "*summary"
      ]
    }
  }
}

Querying "galactic moon" will get us both Jira objects and the documents we indexed:

{
  "took": 3,
  "timed_out": false,
  "_shards": {
    "total": 3,
    "successful": 3,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 3,
      "relation": "eq"
    },
    "max_score": 1.2613049,
    "hits": [
      {
        "_index": "bank",
        "_id": "Marketing Mars-MM-2",
        "_score": 1.2613049,
        "_source": {
          "Type": "Task",
          "Custom_Fields": {
            "satisfaction": null,
            "Approvals": null,
            "Change reason": null,
            "Epic Link": null,
            "Actual end": null,
            "Design": null,
            "Campaign assets": null,
            "Department": null,
            "story point estimate": null,
            "Approver groups": null,
            "[CHART] Date of First Response": null,
            "Request Type": null,
            "Campaign goals": null,
            "Project overview key": null,
            "Related projects": null,
            "Campaign type": null,
            "Impact": null,
            "Request participants": [],
            "Locked forms": null,
            "Time to first response": null,
            "Work category": null,
            "Audience": null,
            "Open forms": null,
            "Details": null,
            "sprint": null,
            "stakeholders": null,
            "Marketing asset type": null,
            "submitted forms": null,
            "start date": null,
            "Actual start": null,
            "Category": null,
            "Change risk": null,
            "Target start": null,
            "Issue color": null,
            "Parent Link": {
              "hasEpicLinkFieldDependency": false,
              "showField": false,
              "nonEditableReason": {
                "reason": "EPIC_LINK_sHOULD_BE_UsED",
                "message": "To set an epic as the parent, use the epic link instead"
              }
            },
            "Format": null,
            "Target end": null,
            "Approvers": null,
            "Team": null,
            "Change type": null,
            "satisfaction date": null,
            "Request language": null,
            "Amount": null,
            "Rank": "0|i0003j:",
            "Affected services": null,
            "Type": null,
            "Time to resolution": null,
            "Total forms": null,
            "[CHART] Time in status": null,
            "Organizations": [],
            "Flagged": null,
            "Project overview status": null
          },
          "Issue": {
            "statuscategorychangedate": "2024-11-01T17:52:30.550-0300",
            "issuetype": {
              "avatarId": 10318,
              "hierarchyLevel": 0,
              "name": "Task",
              "self": "https://xxxx.atlassian.net/rest/api/2/issuetype/10017",
              "description": "A small, distinct piece of work.",
              "entityId": "f30ea676-7b3d-44ad-9858-558081742a2e",
              "id": "10017",
              "iconUrl": "https://xxxx.atlassian.net/rest/api/2/universal_avatar/view/type/issuetype/avatar/10318?size=medium",
              "subtask": false
            },
            "components": [],
            "timespent": null,
            "timeoriginalestimate": null,
            "project": {
              "simplified": true,
              "avatarUrls": {
                "48x48": "https://xxxx.atlassian.net/rest/api/2/universal_avatar/view/type/project/avatar/10418",
                "24x24": "https://xxxx.atlassian.net/rest/api/2/universal_avatar/view/type/project/avatar/10418?size=small",
                "16x16": "https://xxxx.atlassian.net/rest/api/2/universal_avatar/view/type/project/avatar/10418?size=xsmall",
                "32x32": "https://xxxx.atlassian.net/rest/api/2/universal_avatar/view/type/project/avatar/10418?size=medium"
              },
              "name": "Marketing Mars",
              "self": "https://xxxx.atlassian.net/rest/api/2/project/10003",
              "id": "10003",
              "projectTypeKey": "business",
              "key": "MM"
            },
            "description": null,
            "fixVersions": [],
            "aggregatetimespent": null,
            "resolution": null,
            "timetracking": {},
            "security": null,
            "aggregatetimeestimate": null,
            "attachment": [],
            "resolutiondate": null,
            "workratio": -1,
            "summary": "Conquer the moon",
            "issuerestriction": {
              "issuerestrictions": {},
              "shouldDisplay": true
            },
            "watches": {
              "self": "https://xxxx.atlassian.net/rest/api/2/issue/MM-2/watchers",
              "isWatching": true,
              "watchCount": 1
            },
            "lastViewed": "2024-11-01T17:52:34.925-0300",
            "creator": {
              "accountId": "712020:88983800-6c97-469a-9451-79c2dd3732b5",
              "emailAddress": "contornan_cliche.0y@icloud.com",
              "avatarUrls": {
                "48x48": "https://secure.gravatar.com/avatar/f098101294d1a0da282bb2388df8c257?d=https%3A%2F%2Favatar-management--avatars.us-west-2.prod.public.atl-paas.net%2Finitials%2FTM-3.png",
                "24x24": "https://secure.gravatar.com/avatar/f098101294d1a0da282bb2388df8c257?d=https%3A%2F%2Favatar-management--avatars.us-west-2.prod.public.atl-paas.net%2Finitials%2FTM-3.png",
                "16x16": "https://secure.gravatar.com/avatar/f098101294d1a0da282bb2388df8c257?d=https%3A%2F%2Favatar-management--avatars.us-west-2.prod.public.atl-paas.net%2Finitials%2FTM-3.png",
                "32x32": "https://secure.gravatar.com/avatar/f098101294d1a0da282bb2388df8c257?d=https%3A%2F%2Favatar-management--avatars.us-west-2.prod.public.atl-paas.net%2Finitials%2FTM-3.png"
              },
              "displayName": "Tomas Murua",
              "accountType": "atlassian",
              "self": "https://xxxx.atlassian.net/rest/api/2/user?accountId=712020%3A88983800-6c97-469a-9451-79c2dd3732b5",
              "active": true,
              "timeZone": "Chile/Continental"
            },
            "subtasks": [],
            "created": "2024-11-01T17:52:30.289-0300",
            "reporter": {
              "accountId": "712020:88983800-6c97-469a-9451-79c2dd3732b5",
              "emailAddress": "contornan_cliche.0y@icloud.com",
              "avatarUrls": {
                "48x48": "https://secure.gravatar.com/avatar/f098101294d1a0da282bb2388df8c257?d=https%3A%2F%2Favatar-management--avatars.us-west-2.prod.public.atl-paas.net%2Finitials%2FTM-3.png",
                "24x24": "https://secure.gravatar.com/avatar/f098101294d1a0da282bb2388df8c257?d=https%3A%2F%2Favatar-management--avatars.us-west-2.prod.public.atl-paas.net%2Finitials%2FTM-3.png",
                "16x16": "https://secure.gravatar.com/avatar/f098101294d1a0da282bb2388df8c257?d=https%3A%2F%2Favatar-management--avatars.us-west-2.prod.public.atl-paas.net%2Finitials%2FTM-3.png",
                "32x32": "https://secure.gravatar.com/avatar/f098101294d1a0da282bb2388df8c257?d=https%3A%2F%2Favatar-management--avatars.us-west-2.prod.public.atl-paas.net%2Finitials%2FTM-3.png"
              },
              "displayName": "Tomas Murua",
              "accountType": "atlassian",
              "self": "https://xxxx.atlassian.net/rest/api/2/user?accountId=712020%3A88983800-6c97-469a-9451-79c2dd3732b5",
              "active": true,
              "timeZone": "Chile/Continental"
            },
            "aggregateprogress": {
              "total": 0,
              "progress": 0
            },
            "priority": {
              "name": "Medium",
              "self": "https://xxxx.atlassian.net/rest/api/2/priority/3",
              "iconUrl": "https://xxxx.atlassian.net/images/icons/priorities/medium.svg",
              "id": "3"
            },
            "labels": [],
            "environment": null,
            "timeestimate": null,
            "aggregatetimeoriginalestimate": null,
            "versions": [],
            "duedate": null,
            "progress": {
              "total": 0,
              "progress": 0
            },
            "issuelinks": [],
            "votes": {
              "hasVoted": false,
              "self": "https://xxxx.atlassian.net/rest/api/2/issue/MM-2/votes",
              "votes": 0
            },
            "comment": {
              "total": 0,
              "comments": [],
              "maxResults": 0,
              "self": "https://xxxx.atlassian.net/rest/api/2/issue/10018/comment",
              "startAt": 0
            },
            "assignee": null,
            "worklog": {
              "total": 0,
              "maxResults": 20,
              "startAt": 0,
              "worklogs": []
            },
            "updated": "2024-11-01T17:52:42.711-0300",
            "status": {
              "name": "To Do",
              "self": "https://xxxx.atlassian.net/rest/api/2/status/10014",
              "description": "",
              "iconUrl": "https://xxxx.atlassian.net/",
              "id": "10014",
              "statusCategory": {
                "colorName": "blue-gray",
                "name": "To Do",
                "self": "https://xxxx.atlassian.net/rest/api/2/statuscategory/2",
                "id": 2,
                "key": "new"
              }
            }
          },
          "id": "Marketing Mars-MM-2",
          "_timestamp": "2024-11-01T17:52:42.711-0300",
          "Key": "MM-2",
          "_allow_access_control": [
            "account_id:712020:88983800-6c97-469a-9451-79c2dd3732b5",
            "name:Tomas-Murua"
          ]
        }
      },
      {
        "_index": "galactic_documents",
        "_id": "2",
        "_score": 0.61183906,
        "_source": {
          "document_id": "GBFF-002",
          "title": "User Guide for the Galactic Banking App",
          "content": "Welcome to the Galactic Banking application by Interstellar Finance Corp. Here you can: Transfer galactic credits to any registered account across the Milky Way. Check your balance and manage your investments in real-time. Access interplanetary loans with ease. For your security, use multi-factor authentication each time you log in.",
          "release_date": "2024-01-01",
          "page_count": 3,
          "tags": [
            "user guide",
            "application"
          ]
        }
      },
      {
        "_index": "galactic_documents",
        "_id": "3",
        "_score": 0.5029222,
        "_source": {
          "document_id": "GBFF-003",
          "title": "Technical specifications Report - Galactic Banking Project",
          "content": "This report details the technical architecture of the Galactic Banking application: Microservices-based backend for scalability and performance. secure communication protocols utilizing quantum encryption. Transaction management adapted to environments with gravity variations and time dilation.",
          "release_date": "2024-01-01",
          "page_count": 7,
          "tags": [
            "technical",
            "specifications",
            "architecture"
          ]
        }
      }
    ]
  }
}

If a document is too long, you can add the option _source to the query to only include the fields that you need. If you just want to remove some fields, we'll cover that option in the second part of this series.

Document Level security (DLs)

We will now configure Document Level security (DLs) to match Jira permissions to the ones in Elasticsearch so that when users search, they can only see what they are allowed to see in Jira.

To begin, we'll go to the connector's Control Panel in Elastic Cloud and click on Access Control sync.

Jira connector's Control Panel in Elastic Cloud

This sync will bring the access and permission info from the Jira users. To test this, I've made another Jira board to which the user "Gustavo" does not have access.

Note: Do not forget to run content sync after creating the board.You can run one time syncs, or schedule based.

Let's begin checking that the documents from the new board are there:

GET bank/_search
{
  "_source": ["Issue.summary"],
  "query": {
   "match": {
     "Issue.project.name": "Marketing Mars"
   }
  }
}

We can effectively see the issues:

{
  "took": 2,
  "timed_out": false,
  "_shards": {
    "total": 2,
    "successful": 2,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 3,
      "relation": "eq"
    },
    "max_score": 0.7473189,
    "hits": [
      {
        "_index": "bank",
        "_id": "Marketing Mars-MM-1",
        "_score": 0.7473189,
        "_source": {
          "Issue": {
            "summary": "Conquer Mars"
          }
        }
      },
      {
        "_index": "bank",
        "_id": "Marketing Mars-MM-3",
        "_score": 0.7473189,
        "_source": {
          "Issue": {
            "summary": "Conquering Earth"
          }
        }
      },
      {
        "_index": "bank",
        "_id": "Marketing Mars-MM-2",
        "_score": 0.7473189,
        "_source": {
          "Issue": {
            "summary": "Conquer the moon"
          }
        }
      }
    ]
  }
}

However, since the user "Gustavo" does not have access, he should not be able to see them.

Let's look for the user's document in the ACL filter index to see their permissions.

GET .search-acl-filter-bank/_search

Response:

{
  "_index": ".search-acl-filter-bank",
  "_id": "63c04b092341bff4fff6e0cb",
  "_score": 1,
  "_source": {
    "created_at": "2024-11-01T23:19:35.784996+00:00",
    "id": "63c04b092341bff4fff6e0cb",
    "_timestamp": "2024-11-01T05:42:04.410478+00:00",
    "identity": {
      "account_id": "account_id:63c04b092341bff4fff6e0cb",
      "email_address": null,
      "display_name": "name:Gustavo",
      "locale": "locale:en_Us"
    },
    "query": {
      "template": {
        "source": """{
          "bool": {
            "should": [
              {
                "bool": {
                  "must_not": {
                    "exists": {
                      "field": "_allow_access_control"
                    }
                  }
                }
              },
              {
                "terms": {
                  "_allow_access_control.enum": {{#toJson}}access_control{{/toJson}}
                }
              }
            ]
          }
        }""",
        "params": {
          "access_control": [
            "account_id:63c04b092341bff4fff6e0cb",
            "group_id:d3f28403-7e99-4262-8f11-77a75bcd33d8",
            "role_key:jira-software"
          ]
        }
      }
    }
  }
}

This index includes the user id and all of their Jira groups. We need to make a match between the content in the user's access control and the field _allowed_access_control in each document.

We'll create an API Key for Gustavo using the command below. You must copy the query.template value from the previous step:

POsT /_security/api_key
{
  "name": "gustavo",
  "expiration": "30d",
  "role_descriptors": {
    "jira-role": {
      "index": [
        {
          "names": [
            "bank",
            "galactic_documents"
          ],
          "privileges": [
            "read",
            "view_index_metadata"
          ],
          "query": {
            "template": {
              "params": {
                "access_control": [
                  "account_id:63c04b092341bff4fff6e0cb",
                  "group_id:d3f28403-7e99-4262-8f11-77a75bcd33d8",
                  "role_key:jira-software"
                ]
              },
              "source": """{
                "bool": {
                  "should": [
                    {
                      "bool": {
                        "must_not": {
                          "exists": {
                            "field": "_allow_access_control"
                          }
                        }
                      }
                    },
                    {
                      "terms": {
                        "_allow_access_control.enum": {{#toJson}}access_control{{/toJson}}
                      }
                    }
                  ]
                }
              }"""
            }
          }
        }
      ]
    }
  }
}

Note that we're only giving access to the indices in this article through this option.

The response for the creation of the API Key for Gustavo is this:

{
  "id": "yLa1FJMBU4bZPaw5stnl",
  "name": "gustavo",
  "expiration": 1733811245816,
  "api_key": "UrGdsnDFsyGxjQvLayw5jQ",
  "encoded": "eUxhMUZKTUJVNGJaUGF3NVN0bmw6VXJHZHNuREZTeUd4alF2TGF5dzVqUQ=="
}

You can use curl to test that we can run searches using the API KEY and it won't bring info from the Marketing board, since Gustavo does not have access to it.

curl --location --request GET 'https://interstellar-finance-corp.es.us-central1.gcp.cloud.es.io/bank/_search' \
--header 'Authorization: ApiKey eUxhMUZKTUJVNGJaUGF3NVN0bmw6VXJHZHNuREZTeUd4alF2TGF5dzVqUQ==' \
--header 'Content-Type: application/json' \
--data '{
  "_source": ["Issue.summary"],
  "query": {
   "match": {
     "Issue.project.name": "Marketing Mars"
   }
  }
}'

Response:

{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 0,
      "relation": "eq"
    },
    "max_score": null,
    "hits": []
  }
}

We can see that Gustavo did not get any info since he did not have access. Now, let's test with the documents from the board that he is allowed to see:

curl --location --request GET 'https://interstellar-finance-corp.es.us-central1.gcp.cloud.es.io/bank/_search?pretty=true' \
--header 'Authorization: ApiKey eUxhMUZKTUJVNGJaUGF3NVN0bmw6VXJHZHNuREZTeUd4alF2TGF5dzVqUQ==' \
--header 'Content-Type: application/json' \
--data '{
  "_source": ["Issue.summary"],
  "query": {
   "match": {
     "Issue.project.name": "Galactic Banking Project"
   }
  }
}'

Response:

{
  "took" : 7,
  "timed_out" : false,
  "_shards" : {
    "total" : 2,
    "successful" : 2,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 3,
      "relation" : "eq"
    },
    "max_score" : 3.1784885,
    "hits" : [
      {
        "_index" : "bank",
        "_id" : "Galactic Banking Project-GBP-3",
        "_score" : 3.1784885,
        "_source" : {
          "Issue" : {
            "summary" : "Intergalactic security and Compliance"
          }
        }
      },
      {
        "_index" : "bank",
        "_id" : "Galactic Banking Project-GBP-2",
        "_score" : 0.5469647,
        "_source" : {
          "Issue" : {
            "summary" : "Bank Application Frontend"
          }
        }
      },
      {
        "_index" : "bank",
        "_id" : "Galactic Banking Project-GBP-1",
        "_score" : 0.5469647,
        "_source" : {
          "Issue" : {
            "summary" : "Development of API for International Transfers"
          }
        }
      }
    ]
  }
}

Conclusion

As you can see, integrating Elasticsearch with Jira has many benefits, like being able to get a unified search on all the projects you're working on as well as being able to run more advanced searches in more than one data source. The added DLs is a quick and easy way to guarantee that users will maintain the access they already had in the original sources.

Want to get Elastic certified? Find out when the next Elasticsearch Engineer training is running!

Elasticsearch is packed with new features to help you build the best search solutions for your use case. Dive into our sample notebooks to learn more, start a free cloud trial, or try Elastic on your local machine now.