Databricks

Exchange data between Databricks and JustAI: send your engagement and conversion data to JustAI, and receive a record of what JustAI served for your own analysis. Share this page with your data team.

Send your data to JustAI

JustAI can count engagement and custom conversion events exported from your Databricks workspace, so they show up in your experiment results. Your team sets up the export job, and JustAI provides a shared S3 bucket to read from.

To enable this flow:

• Set up a Databricks export job that writes to the shared S3 bucket provided by JustAI.
• Share your Databricks workspace details so we can validate the setup.

If you use Iterable, the Iterable warehouse export guide walks through the full export recipe, including sample Spark SQL.

Receive JustAI data in Databricks

JustAI can export a record of which content each user received — variant metadata and message-level send records — into your Databricks environment using a secure, shared S3 bucket (recommended) or Delta Sharing. This lets you join JustAI variants with your internal engagement, attribution, LTV, and conversion models inside Databricks.

JustAI will write daily exports containing:

1. Variant metadata (copy_id → template_id, vars, attrs)
2. Message-level send records (event_timestamp, message IDs, copy_id)

You will pull these into Databricks using:

• A Databricks-accessible S3 bucket provided by your team (recommended), or
• Delta Sharing to receive a table directly (alternative).

Option A (recommended) — S3 export into your Databricks workspace

Your team

1) Create a shared S3 bucket or prefix for JustAI

You may use an existing bucket; JustAI only needs a dedicated prefix.

2) Create an IAM role that JustAI can assume

Minimum permissions:

• s3:ListBucket
• s3:GetObject
• s3:PutObject
• s3:DeleteObject

Example pattern:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowListingOfExportPrefix",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::<CLIENT_BUCKET_NAME>",
      "Condition": {
        "StringLike": {
          "s3:prefix": "<YOUR_EXPORT_PREFIX>/*"
        }
      }
    },
    {
      "Sid": "AllowReadWriteDeleteOnExportObjects",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Resource": "arn:aws:s3:::<CLIENT_BUCKET_NAME>/<YOUR_EXPORT_PREFIX>/*"
    }
  ]
}

3) Add bucket policy to allow the JustAI role

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowJustAIList",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::<JUSTAI_AWS_ACCOUNT_ID>:role/<JUSTAI_EXPORT_ROLE_NAME>"
      },
      "Action": [
        "s3:ListBucket"
      ],
      "Resource": "arn:aws:s3:::<CLIENT_BUCKET_NAME>",
      "Condition": {
        "StringLike": {
          "s3:prefix": "<YOUR_EXPORT_PREFIX>/*"
        }
      }
    },
    {
      "Sid": "AllowJustAIObjectRW",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::<JUSTAI_AWS_ACCOUNT_ID>:role/<JUSTAI_EXPORT_ROLE_NAME>"
      },
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Resource": "arn:aws:s3:::<CLIENT_BUCKET_NAME>/<YOUR_EXPORT_PREFIX>/*"
    }
  ]
}

4) Load the Parquet exports into Databricks

Using either:

• Databricks Autoloader
• Scheduled notebook
• Delta Live Tables pipeline
• Custom Spark job

JustAI

1) Provide IAM Role ARN

JustAI provides a role for your bucket to whitelist.

2) Produce daily (or hourly) exports

Written to: s3://<client-bucket>/justai/exports/YYYY/MM/DD/HH/

• Format: Parquet, partitioned by date/hour
• Backfills run ad hoc and overwrite existing partitions

3) Write variant metadata tables

JustAI writes a companion file containing:

• copy_id (JustAI copy ID)
• template_id (JustAI template ID)
• vars (subject/preheader/body/etc.)
• attrs (persona, etc.)

4) Write message-level records

• copy_id (JustAI copy ID)
• template_id (JustAI template ID)
• event_timestamp (Unix timestamp)
• message_id (Iterable Message ID)
• itbl_campaign_id (Iterable campaign ID)
• itbl_template_id (Iterable template ID)

Data structures

Message send events (Parquet rows)

{
  "event_timestamp": 1736387200,
  "user_id": "abc123",
  "message_id": "000000000",
  "copy_id": "6b3f2dd3-1c57-4f56-bc26-89af7bb6cb30"
}

Variant metadata

{
  "copy_id": "6b3f2dd3-1c57-4f56-bc26-89af7bb6cb30",
  "template_id": "welcome_email_1",
  "vars": {
    "subject": "Welcome to our Community",
    "preheader": "Let’s get started!",
    "body": "<html>..."
  },
  "attrs": {
    "persona": "learner",
    "age": "18-24"
  }
}

Option B — Delta Sharing (Databricks native)

How it works

1. Your team creates a share in Unity Catalog
2. JustAI publishes tables into that share:
   • justai.send_events
   • justai.copy_variants

Your team reads them directly in Databricks using:

SELECT * FROM delta.`/shares/justai/send_events`

This avoids S3 roles entirely but requires Unity Catalog.

End-to-end workflow summary

Your team

1. Create S3 bucket + prefix
2. Create IAM role with Put/Get/List
3. Add bucket policy for JustAI role
4. Pull data into Databricks (Autoloader / DLT / Spark job)

JustAI

1. Provide IAM role ARN
2. Produce hourly/daily Parquet exports (partitioned)
3. Export variant metadata table
4. Export send events table
5. Perform backfills on request

Example Databricks Autoloader setup

df = (spark.readStream
      .format("cloudFiles")
      .option("cloudFiles.format", "parquet")
      .load("s3://<bucket>/justai/exports/"))