Exporting payments data
Export your payments data to Redshift from Hyperswitch
Last updated
Export your payments data to Redshift from Hyperswitch
Last updated
In this section, you would be able to understand how you can export your payments data to Redshift from Hyperswitch, the architecture behind it, schema and the queries.
Exporting your payments data to Amazon Redshift enhances analytics by leveraging Redshift's high-performance query capabilities. This allows for efficient data analysis, reporting, and business intelligence there by deriving valuable insights
Prerequisite: You need to have an AWS account with redshift enabled. More details can be found here https://aws.amazon.com/redshift/
You are required to create a new IAM role for Redshift use and provide Hyperswitch with the corresponding role ARN. This IAM role must be configured with S3 read permissions.
After sharing the ARN with Hyperswitch, We will share the S3 bucket & path that is to be synced for data along with providing access to the IAM role from where you will be able to get files from the S3
Once the above step is done, you need to create the table schema on Redshift
Post which you can proceed with the below
Handle the ingestion & post processing of data using scripts
(OR)
The files will be plain csv files with 1st row being a header
The file path would look be s3://<bucket>/<merchant_id>/<version>/<payments>/<date>.csv
There will be one csv file corresponding to each day upto 7 days. Updates to the payments data will be in-place
Changes to file formats, content or likewise would change the version in the above path and would be communicated.
Data update schedule | 6 hours frequency up to 7 days |
Data retention on S3 folder | 7 days |
Type of data exposed | |
Data storage location | us-east-1 |
Redshift supports ingesting csv data directly from S3 files which we’ll rely on.
The ingestion to redshift would happen via a COPY job, this can be automated via the following options
You can use the auto copy job if running a preview cluster
Or the more mainstream lambda loader
The above query creates a temporary table to load all the csv data & then merges this with the main table while deduplicating based on payment_id
payments as per