[System Design] Data Migration

LiveRunGrow
6 min readSep 8, 2023
Online image

Problem Description

Multi-tenant online system; Migrate the data of some tenants to another data storage securely and with stability.

Step 1: Gather Requirements

First, we need to ask the interviewer some questions to clarify the requirements. Lets imagine the answers to those questions are in italics (If it sounds strange, it’s because I crafted them from my own imagination).

Functional

  • What kind of data do we want to migrate? User profile/account data
  • What’s the source and target schema? The schemas are the same.
  • What’s the source and target system? SQL Databases. They are the same, except hosted in different servers.
  • What kind of data is classified under the special tenant category and needs to be migrated? The purpose of the data migration project is to separate data belonging to users in different geographic regions to different servers. For example, we want to only move US Tiktok accounts to storage servers in the US.

Non functional

  • Does the system always need to be available? Can system downtime be accepted? System downtimes are not accepted.
  • Are we concerned about the time it takes for the data migration to complete? Does it need to be fast? It does not need to happen in a single day. It can be done over a period of time.
  • If errors happen during the data migration, do we need to make sure it’s recoverable? Yes, important not to lose data.
  • If data is changed during the migration process, we need to capture them. What kind of data changes may happen? Delete Insert Update. For simplicity, no update to location happens.

From the answers, we get a better idea of what the data migration is for and how it needs to be done.

Step 2: Estimation

Traffic (QPS)

We can consider the formula below to come up with an estimated QPS. To be honest, the accuracy of this is not important. I think it’s just to demonstrate your number sense and that you know that the amount of data involved is big.

--

--

LiveRunGrow

𓆉︎ 𝙳𝚛𝚎𝚊𝚖𝚎𝚛 🪴𝙲𝚛𝚎𝚊𝚝𝚘𝚛 👩‍💻𝚂𝚘𝚏𝚝𝚠𝚊𝚛𝚎 𝚎𝚗𝚐𝚒𝚗𝚎𝚎𝚛 ☻ I write & reflect weekly about software engineering, my life and books. Ŧ๏ɭɭ๏ฬ ๓є!