How to transfer Slack data to S3 using AWS appflow, query using athena and visualise in quicksight

Abdul Jaleel Kavungal
8 min readFeb 16, 2021

Introduction to AWS Appflow, Athena and Quicksight

  • Amazon AppFlow is a fully managed no-code integration service that helps to transfer data between Software-as-a-Service (SaaS) applications like Salesforce, Marketo, Slack, and ServiceNow, and AWS services like Amazon S3 and Amazon Redshift, in just a few clicks.
  • Amazon Athena is a serverless, interactive query service to query data and analyzes big data in Amazon S3 using standard SQL.
  • Amazon Quicksight is an AWS dashboarding service. It has a user-friendly drag and drop interface to create charts and full dashboards in less than an hour.

Architecture

In this blog post, we will be integrating Slack with Amazon Appflow and transferring data from Slack to Amazon S3. Then we will create Amazon Athena table and query the slack data using SQL. In the end, we will be creating charts in Amazon Quicksight using Amazon Athena as a source for the dataset.

In Amazon Quicksight we will be analysing messages that are posted to Slack #general channel. We will mockup messages so that the messages will look like CI/CD pipeline deployment notifications.

Note: Use ap-southeast-2 as region for all the resources in this workshop.

Steps

  • Step 1: Preliminary Setup
  • Step 2: Create and install a Slack App
  • Step 3: Connect Slack to AppFlow and create a flow
  • Step 4: Run the AppFlow
  • Step 5: Create Athena Table & Query data in S3
  • Step 6: Create Amazon Quicksight Charts

Preliminary Setup

As a pre-requisite for the tasks explained in this post, we will be using Amazon S3 buckets for below purposes:

  • Datalake for storing Slack data
  • Saving Athena query results

These S3 buckets must be created prior to starting the workshop.

Create S3 Bucket for storing Slack data

Abdul Jaleel Kavungal

No one is ready for the future, the best way to predict the future is to BUILD it