I just completed some work for a client where I had to migrate all their emails from Help Scout to Gmail (Google Mail). Turns out this wasn’t as easy as I’d hoped. There’s very little information that I could find on how to do this so I thought I write it down here. (BTW, I tried reaching out to a couple different company’s for quotes on this. Their price was around 1–2k for an existing connections. Since they didn’t have connections already built, the only solution was a custom one which who knows how much that’ll be).
The emails I was migrating from Help Scout to Gmail were only in Help Scout, meaning, they weren’t being sent via Gmail to begin with; the only location they existed was in Help Scout. This means I couldn’t just use the built-in migration service that Gmail offers.
APIs
Help Scout offers an OAuth 2 API to access your data, but it’s actually built for app developers in mind first. Since I wasn’t in the market for building an app with Help Scout, I needed to obtain an API key just for making requests myself. Buried at the bottom of their authentication documentation they do show you how you can get a temporary key without having to build an OAuth app.
Gmail also offers an API but again it’s OAuth. Again you can get credentials to make API calls without having to build an entire OAuth flow. I won’t write down how to get the credentials here because each case will be different and Google provides a bit of information on how you can do that.
Downloading
Once I had the keys and credentials, I started downloading all the emails from Help Scout using their API. Help Scout has two words they use in their API, conversations and threads. Conversations as I understand it are a grouping of threads between one individual. Threads are the individual emails and action items taken during a conversation. Since I was just concerned with the email part of this migration, I had to weed out all the emails from the threads and import from there.
Formatting
Gmail has two ways of getting emails into an inbox via their API without actually sending the email. One is import and the other insert and they have slight differences. I chose insert for this project. The tricky part is the formatting required by Gmail is an RFC 2822 formatted email. Help Scout will give you the RFC 2822 format only for incoming emails, not outgoing. Since I was trying to get everything, this wasn’t going to work. That meant I’d have to build an RFC 2822 formatted email myself. I could get enough information from the Help Scout API to build it but then there was the issue of email attachments.
Attachments
Attachments can be done one of two ways; either in the body of the email or separately uploaded. The documentation around attachments and Gmail is somewhat vague in my opinion. They will tell you how to upload attachments but don’t exactly specify how to attach them to emails (especially during inserts). I ended up contacting Google support and after almost a week of back and forth, they couldn’t give me an answer. I finally an obscure article that covered an example of using attachments in the body of the email and I was able to fit it to my specific use case. This worked up until I hit attachments bigger than about 10mb. The issue was the request was timing out because of trying to upload the entire payload. I came up with a temporary solution but I still never figured out the proper way Gmail says to do it. In the end it worked however.
Uploading/Inserting
I first tested on a temporary account to make sure the data I was migrating was correct and in the right order. Then I ran the download first, saving the individual threads and attachments locally. This takes some time and I didn’t want a break during the migration so saving locally helps mitigate that issue. Then I ran the Gmail insert. Not much issue here, however I did save progress as I went along so if I had and error or crash and had to restart, I could pickup from where I left off.
And that pretty much sums it up. Once the migration is complete and you’ve verified your data is moved over, you can choose to keep or close the Help Scout mailbox.
Notes
- Gmail API only works with RFC 2822 formatted email messages. Help Scout only provides that format for incoming messages, not outgoing. I couldn’t come up with a solution that didn’t involve making my own RFC 2822 format.
- The Gmail API lacks examples on how to insert/import emails with multiple attachments. I even contacted their support and they couldn’t provide me with a proper example.
- Threads are not done automatically by Gmail. You need to generate a unique message ID for each email and keep track of the first email on that thread that was inserted. Then for each subsequent email, pass the threadId param along with the payload. Also the RFC formatted email needs to include the proper message-id, in-reply-to and references headers.
- I did not get into forward chaining. This can start to get quite complicated quickly and it wasn’t an absolute need to client to preserve this so I left it out. Forwards were treated as replies to an email. This ended up working fine.
- Dates in Gmail are expected in a certain way, different than the RFC spec. I don’t agree with it but I don’t really have a say in how they want it to work.
- Certain things in the Help Scout API are “assumed”, meaning there’s no documentation on it and you just have to assume that’s the way it is. An example of this is on the outgoing emails, there’s no “from” email. That’s because the from is the account/mailbox that’s sending the email. This added quite a learning curve to using the API.
- This was only needed because the emails weren’t being sent through Gmail as a provider but instead solely via Help Scout. All I can say is it’s a good thing Help Scout had an API otherwise this could’ve been a lot harder.
If you’re interested in migrating your data from Help Scout to Gmail and aren’t ready to implement your own solution through APIs, feel free to reach out.
