Notifications

Notifications (Projects > your_project > Project Settings > Notifications ) are events to which you can subscribe. When they are triggered, they deliver a message to an external target system such as emails, Amazon SQS or SNS systems or HTTP post requests. The following table describes available system events to subscribe to:

When you subscribe to overlapping event codes such as ICA_EXEC_002 (analysis success) and ICA_EXEC_028 (analysis status change) you will get both notifications when analysis success occurs.

Delivery Targets

Event notifications can be delivered to the following delivery targets:

Amazon Resource Policy Settings

In order to allow the platform to deliver events to Amazon SQS or SNS delivery targets, a cross-account policy needs to be added to the target Amazon service.

{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Effect":"Allow",
         "Principal":{
            "AWS":"arn:aws:iam::<platform_aws_account>:root"
         },
         "Action":"<action>",
         "Resource": "<arn>"
      }
   ]
}

Substitute the variables in the example above according to the table below.

See examples for setting policies in Amazon SQS and Amazon SNS

Amazon SNS Topic

To create a subscription to deliver events to an Amazon SNS topic, one can use either GUI or API endpoints.

To create a subscription via GUI, select Projects > your_project > Project Settings > Notifications and then select New ICA subscription. Then select an event from the dropdown menu, insert optional filter, select the channel type (SNS), and then insert the ARN from the target SNS topic and the AWS region.

To create a subscription via API, use the endpoint /api/notificationChannel to create a channel and then /api/projects/{projectId}/notificationSubscriptions to create a notification subscription.

Amazon SQS Queue

To create a subscription to deliver events to an Amazon SQS queue, you can use either GUI or API endpoints.

To create a subscription via the GUI, select Projects > your_project > Project Settings > Notifications, then select New ICA subscription. Next, select an event from the dropdown menu, choose SQS as the way to receive the notifications, enter your SQS URL, and if applicable for that event, choose a payload version. Not all payload versions are applicable for all events and targets, so the system will filter the options out for you. Finally, you can enter a filter expression to filter which events are relevant for you. Only those events matching the expression will be received.

To create a subscription via API, use the endpoint /api/notificationChannel to create a channel and then /api/projects/{projectId}/notificationSubscriptions to create a notification subscription.

Messages delivered to AWS SQS contain the following event body attributes:

The following example is a Data Updated event payload sent to an AWS SQS delivery target (condensed for readability):

{
    "correlationId": "2471d3e2-f3b9-434c-ae83-c7c7d3dcb4e0",
    "timestamp": "2022-10-06T07:51:09.128Z",
    "eventCode": "ICA_DATA_100",
    "description": "Data updates",
    "payload": {
        "id": "fil.8f6f9511d70e4036c60908daa70ea21c",
        ...
    }
}

Filtering

Notification subscriptions will trigger for all events matching the configured event type. A filter may be configured on a subscription to limit the matching strategy to only those event payloads which match the filter.

The filter expressions leverage the JsonPath library for describing the matching pattern to be applied to event payloads. The filter must be in the format [?(<expression>)].

Examples

The Analysis Success event delivers a JSON event payload matching the Analysis data model (as output from the API to retrieve a project analysis).

{
  "id": "0c2ed19d-9452-4258-809b-0d676EXAMPLE",
  "timeCreated": "2021-09-20T12:23:18Z",
  "timeModified": "2021-09-20T12:43:02Z",
  "ownerId": "15d51d71-b8a1-4b38-9e3d-74cdfEXAMPLE",
  "tenantId": "022c9367-8fde-48fe-b129-741a4EXAMPLE",
  "reference": "210920-1-CopyToolDev-9d78096d-35f4-47c9-b9b6-e0cbcEXAMPLE",
  "userReference": "210920-1",
  "pipeline": {
    "id": "20261676-59ac-4ea0-97bd-8a684EXAMPLE",
    "timeCreated": "2021-08-25T01:49:41Z",
    "timeModified": "2021-08-25T01:49:41Z",
    "ownerId": "15d51d71-b8a1-4b38-9e3d-74cdfEXAMPLE",
    "tenantId": "022c9367-8fde-48fe-b129-741a4EXAMPLE",
    "code": "CopyToolDev",
    "description": "CopyToolDev",
    "language": "CWL",
    "pipelineTags": {
      "technicalTags": ["Demo"]
    }
  },
  "status": "SUCCEEDED",
  "startDate": "2021-09-20T12:23:21Z",
  "endDate": "2021-09-20T12:43:00Z",
  "summary": "",
  "finishedSteps": 0,
  "totalSteps": 1,
  "tags": {
    "technicalTags": [],
    "userTags": [],
    "referenceTags": []
  }
}

The below examples demonstrate various filters operating on the above event payload:

• Filter on a pipeline, with a code that starts with ‘Copy’. You’ll need a regex expression for this:

  [?($.pipeline.code =~ /Copy.*/)]

• Filter on status (note that the Analysis success event is only emitted when the analysis is successful):

  [?($.status == 'SUCCEEDED')]

  Both payload Version V3 and V4 guarantee the presence of the final state (SUCCEEDED, FAILED, FAILED_FINAL, ABORTED) but depending on the flow (so not every intermediate state is guaranteed):

  • V3 can have REQUESTED - IN_PROGRESS - SUCCEEDED

  • V4 can have the status REQUESTED - QUEUED - INITIALIZING - PREPARING_INPUTS - IN_PROGRESS - GENERATING_OUTPUTS - SUCCEEDED

• Filter on pipeline, having a technical tag “Demo":

  [?('Demo' in $.pipeline.pipelineTags.technicalTags)]

• Combination of multiple expressions using &&. It's best practice to surround each individual expression with parentheses:

  [?(($.pipeline.code =~ /Copy.*/) && $.status == 'SUCCEEDED')]

Examples for other events

• Filtering ICA_DATA_104 on owning project name. The top level keys on which you can filter are under the payload key, so payload is not included in this filter expression.

  [?($.details.owningProjectName == 'my_project_name')]

Custom Events

Custom events enable triggering notification subscriptions using event types beyond the system-defined event types. When creating a custom subscription, a custom event code may be specified to use within the project. Events may then be sent to the specified event code using a POST API with the request body specifying the event payload.

Currently custom events can only be defined using the API. In order to create a custom event for your project please follow the steps below:

1. Create a new custom event POST {ICA_URL}/ica/rest/api/projects/{projectId}/customEvents a. Your custom event code must be 1-20 characters long, e.g. 'ICA_CUSTOM_123'. b. That event code will be used to reference that custom event type.\

2. Create a new notification channel POST {ICA_URL}/ica/rest/api/notificationChannels a. If there is already a notification channel created with the desired configuration within the same project, it is also possible to get the existing channel ID using the call GET {ICA_URL}/ica/rest/api/notificationChannels.\

3. Create a notification subscription POST {ICA_URL}/ica/rest/api/projects/{projectId}/customNotificationSubscriptions. a. Use the event code created in step 1. b. Use the channel ID from step 2.\

Once the steps above have been completed successfully, the call from the first step POST {ICA_URL}/ica/rest/api/projects/{projectId}/customEvents could be reused with the same event code to continue sending events through the same channel and subscription.

Following is a sample Python function used inside an ICA pipeline to post custom events for each failed metric:

def post_custom_event(metric_name: str, metric_value: str, threshold: str, sample_name: str):
    api_url = f"{ICA_HOST}/api/projects/{PROJECT_ID}/customEvents"
    headers = {
        "Content-Type": "application/vnd.illumina.v3+json",
        "accept": "application/vnd.illumina.v3+json",
        "X-API-Key": f"{ICA_API_KEY}"
    }
    content = {\"code\": \"ICA_CUSTOM_123\", \"content\": { \"metric_name\": metric_name, \"metric_value\": metric_value,\"threshold\": threshold, \"sample_name\": sample_name}}
    json_data = json.dumps(content)
    response = requests.post(api_url, data=json_data, headers=headers)

    if response.status_code != 204:
        print(f"[EVENT-ERROR] Could not post metric failure event for the metric {metric_name} (sample {sample_name}).")