- DATE:
- AUTHOR:
- The Eightfold team
Data infrastructure transition: Datahub migrating from Redshift to StarRocks
Default Off(Config implementation required)
Who: BI Analyst
Requirement: Once these changes are live for your group_id, any new queries written or scheduled in DataHub must follow StarRocks syntax.
Why we're making this change: As data volumes and concurrent usage on DataHub have grown, our Redshift-based backend has started to hit structural limits: query queuing during peak reporting hours, coupled storage/compute scaling, and batch-oriented data refresh cycles. StarRocks, a modern open-source MPP analytics engine, is purpose-built to remove these bottlenecks — giving DataHub a faster, more concurrent, and more scalable foundation without changing anything about how you use the product.
What's new: We are migrating Datahub from Redshift to StarRocks to resolve latency in data hubs and scheduled reports. This upgrade enables quicker data access, delivering complex datasets in seconds to provide a more responsive self-service experience. We have migrated the scheduled queries from Redshift to StarRocks, ensuring system performance no longer hinders data-driven decisions. This feature is being rolled out to all customers in a phased manner, beginning 22nd June, with general availability expected within 4–6 weeks. If you'd like to migrate earlier, simply raise a support ticket and we'll be happy to prioritize you. A banner will appear on the DataHub screen once it goes live.
Performance & Scale Benefits
Area | Redshift (Before) | StarRocks (After) |
|---|---|---|
Query speed | Vulnerable to queuing at peak load | Vectorized, SIMD-accelerated engine - fast even on large scans/joins: • Median latency: ~2-3× faster across all three workload classes • Tail latency (p90/p99): 3-5× faster - the slow, frustrating outlier queries largely disappear • Complex queries: up to 5-6× faster than Redshift |
Concurrency | Bounded by fixed WLM slots | Built for high concurrency; scales elastically. In internal tests, StarRocks held sub-2.5s worst-case latency at 150 QPS where Redshift degraded even on larger hardware. |
Data freshness | Batch-load cadence | Near real-time capable |
Maintenance | VACUUM/resize windows cause variance | Online scaling, minimal disruption |
Tenant isolation and row-level security are unchanged - each customer still queries only their own data, enforced the same way as today.
How it works: The system's backend is being swapped to the StarRocks engine, and existing scheduled queries are being automatically recoded to ensure compatibility. For new self-service querying, the Data Hub now uses StarRocks SQL syntax, which features only minimal differences from the previous Redshift format.
-- Before (Redshift)
SELECT TOP 100
position_name,
NVL(location, 'Unknown') AS location,
COUNT(*) AS applications,
DATEDIFF(day, created_at, GETDATE()) AS age_days,
TO_CHAR(created_at, 'YYYY-MM-DD') AS created_date,
'apps: ' || COUNT(*) AS label
FROM applications
WHERE status ~ 'active'
GROUP BY position_name, location, created_at;
-- After (StarRocks)
SELECT position_name,
COALESCE(location, 'Unknown') AS location,
COUNT(*) AS applications,
DATEDIFF(NOW(), created_at) AS age_days,
DATE_FORMAT(created_at, '%Y-%m-%d') AS created_date,
CONCAT('apps: ', CAST(COUNT(*) AS STRING)) AS label
FROM applications
WHERE status REGEXP 'active'
GROUP BY position_name, location, created_at
LIMIT 100;Key syntax differences to know
Operation | Redshift | StarRocks |
|---|---|---|
Current timestamp | GETDATE() / SYSDATE | NOW() |
Add interval | DATEADD(day, 7, dt) | DATE_ADD(dt, INTERVAL 7 DAY) |
Date difference (days) | DATEDIFF(day, start, end) | DATEDIFF(end, start) — argument order reversed! |
Date difference (other units) | DATEDIFF(minute, start, end) | TIMESTAMPDIFF(MINUTE, start, end) |
Format date | TO_CHAR(dt, 'YYYY-MM-DD') | DATE_FORMAT(dt, '%Y-%m-%d') |
Parse date | TO_DATE(s, 'YYYY-MM-DD') | STR_TO_DATE(s, '%Y-%m-%d') |
NULL default | NVL(col, 0) | COALESCE(col, 0) |
If-then-else | DECODE(...) | CASE ... WHEN ... END |
Cast | col::int | CAST(col AS INT) |
Concatenate strings | a || b | CONCAT(a, b) |
String length (chars) | LEN(s) | CHAR_LENGTH(s) |
Regex match | s ~ 'pattern' | s REGEXP 'pattern' |
String aggregation | LISTAGG(name, ', ') | GROUP_CONCAT(name SEPARATOR ', ') |
Approx. distinct count | APPROXIMATE COUNT(DISTINCT x) | APPROX_COUNT_DISTINCT(x) |
JSON extraction | JSON_EXTRACT_PATH_TEXT(p, 'a', 'b') | GET_JSON_STRING(p, '$.a.b') |
Top N rows | SELECT TOP 10 ... | ... LIMIT 10 |
Quoted identifiers | "my column" |
|
Common mistakes to avoid:
DATEDIFF - argument order is reversed, hence make sure end is first argument
|| is not string concatenation in StarRocks - Always use CONCAT().
Every :: cast must become CAST(... AS ...).
Double-quoted identifiers become string literals in StarRocks — replace "col name" with
col name.LISTAGG, BOOL_AND, BOOL_OR, NVL2, DECODE, TOP n don't exist in StarRocks — rewrite per the table above.
JSON functions move to JSONPath syntax ($.field.subfield).
What stays the same
Eightfold-curated queries have already been migrated to StarRocks syntax in your instance (No action needed).
SQL editor, customer views, saved/scheduled queries, curated queries, and Scheduled email/SFTP exports all work exactly as before.What You Need to Do
What You Need to Do
Watch for the in-app banner on your DataHub screen confirming your group_id has gone live on StarRocks — don't switch your query syntax before that point, since your instance will still be on Redshift until then.
Once Live, update any custom/self-service SQL queries you've written in DataHub's Editor tab to use StarRocks syntax (see the Key Syntax Differences table above). This applies to any query you write or edit going forward — not queries that already exist and haven't been touched.
Raise a support ticket if you'd like to be prioritized for earlier migration, or if you hit a query that fails after the syntax review above.
Value: Efficiency
