Migration and Consolidation to the Cloud - Challenges and Solutions

In this video you'll learn more about migration and consolidation options when you're moving your legacy data and live mailboxes to the cloud.

Here is a transcript of the video:

Welcome to this recorded presentation, titled: "Migration and Consolidation to the cloud: Challenges and Solutions". 

This presentation was originally given by Daniel Maiworm, who is the Product Director in the Data Management team, at Cloudficient.

Let’s look at what we’re going to cover in this video.

For companies with 10,000 or more employees, or with 100 Tb or more data, and with tight migration deadlines there are two big things to think about.

The first part is data migration.

Think about the types of data that your organization needs for both day-to-day use as well as for legal and compliance purposes. The data is sitting in various systems and needs specialist tools, if you want to move data out of these repositories.

And this brings us to the second part: Where you want to go with certain data types in the future, and what is going to be your target infrastructure?

For many businesses they look at the M365 environment.  Migrating to M365, isn’t always straight forward, especially for industries like finance, legal, energy, and pharmaceutical. These are all highly regulated industries. They are more complicated because they have different requirements and regulations, compared to many other companies.

The big question is:

Is all the data that your organization needs, suitable to be migrated into an M365 mailbox?

Or do you need something more?

Let’s look at Cloudficient.

We started Cloudficient in 2017, but almost everyone in the company have been in the archiving and migration space for decades.

We did all the early Windows based migrations, but we decided the most efficient way to continue migrations in the 2020s and beyond is with a cloud service. We’ve also seen changes in the last few years where companies aren’t just doing a single migration, but rather, due to mergers and acquisitions, they’re restructuring almost constantly. You acquire a company. You divest part of your company. You move into new geographies. And so on. We can’t assume any more that an environment that a company has today is going to be stable.

There is a constant digital transformation.

That’s why we decided our products should be developed as a cloud service and that the projects should be run as a managed service. Businesses are involved in the migration design, but in the end, the migration itself is carried out by our specialist team, 24 by 7. We do this because your company wants the migration to be taken care of, and to have experts to monitor the system to keep the migration going.

We do not just look at moving the data. We look at the whole end to end process of doing a migration. We look at the associated business processes, that would normally be tracked in a spreadsheet and done somewhat manually. We automate many of those and design for you, a customized workflow that executes all the required steps.

But there are some challenges. Let’s look at those.

In a nutshell, anything that you can assign to a user is quite easy to get into M365.

If you have a user mailbox or a PST file, you can just say, “Hey, this is Mike's PST” or “This is Jane's mailbox”, and you just move the data over.

When it is shared content, it suddenly becomes an issue, With journal email too, you basically create one email that was originally shared with maybe 100 or 1,000 people, and all the recipients, all the people who were in a distribution list at that time, show up in the text body of what is called the envelope mail, while the actual mail is just an attachment to that. 

That is something that M365 cannot deal with.

So, we have a lot of customers who say: “We switched on journaling for everyone or for that group of people, how do you take this over into the M365 world?”

Same with voice and conference data, or Lotus Domino data, and other shared mailbox data.

How do you deal with that?

We say, this is not really suitable for a user mailbox.  So where do you put it? You can't put it in a user mailbox. Do you want to put it in a shared mailbox? Do you use a third party solution for this? If you put it in a shared mailbox, the problem is if your legal team ever needs to search for anything, they have to search every shared mailbox.

That adds a lot of time, manual effort and it dilutes the quality of audits and reports, as this data is not assignable to a single user.

This is tricky!

Let’s look into this a bit deeper, starting with ‘suitable’ data.

With suitable data, the first thing you look at are dependencies. Consider what you’d do if you have legacy archiving products like Enterprise Vault, or EMC SourceOne. If you go through the classic steps, you would assign someone who's responsible for user communication. You would also have some preparation of M365. Do you need to adjust send and receive limits? Do you need to adjust other policies before or after migration? How are you going to clean up the source environment?

With most of these tasks you’d end up having a mix of manual steps, and perhaps some scripting.

In the end it's a very complex thing.

The way that we've designed our processes, and we think everyone should design their migration, is to automate as much as possible, including all the user communication.

We can automate a lot of steps including, up to where you clean up and decommission the old environment.

How do we do this? You start by defining certain groups of people. For example, people who are no longer there, these are your leavers. And you can define another group as your regular users. And you might define a third group who are your VIPs. Of course, you can have as many groups as you need.

Then you design a whole process where, for example, you ensure that the multi-factor authentication is rolled out, that you have retention tags assigned to M365, that you have the message sizes set correctly, and in the end, you really have a user that has no remnants of the old environment.

These are all steps in the workflow and can be different for the different groups of users.

If you've designed this, it helps to do a proof of concept. It helps you to get consistent results, and it's one of the things that I believe is crucial. You are trying to get as much automation into the process as humanly possible.

Something that many people aren't aware of is: service protection throttling.

It's Microsoft's way to make sure that what you’re doing on your Microsoft tenant doesn’t disturb your ‘neighbors’, in this case other companies that have mailboxes on that Microsoft cloud server. You've got 150 megabytes in five minutes that you can push into a mailbox, and then you get slowed down. This can dramatically affect the time it takes to do a migration, at least, it has a big impact for some migration software solutions. In simple terms, the way that our software works, is that we take 150 megabytes, go to the next mailbox, take 150 megabytes there, and go to the next mailbox, and so on. Therefore, we never run into throttling, giving you performance numbers that you normally don't see with the classic on-premise products.

If I say we've migrated a Petabyte of data in just over 70 days, that is quite impressive if you know what's normally possible with some of these other on-premise platforms who cannot scale wide, like we do.

Now, let’s look at what we call ‘unsuitable’ data.

If you've got journal data, you should take a moment to think about what you would do if you were to create a new information governance platform. How would I design such a platform to tick all relevant boxes?

You probably would think about low-code or no-code solutions, industry standards like S3, use open source libraries, and that it should be a cloud platform. You’d want it to be scalable using Kubernetes, you’d want auditing and reporting, and so on. You’d think about data formats. Why would you use anything other than EML or MSG or the native file formats?

In the past you had proprietary formats like DVS files with Enterprise Vault, some other platforms might have used some kind of ZIP based storage file. But we think the best way to store the data is in native format. This makes it easier to index the data when needed.

You could use AI with the data to detect anti-money laundering, or anti-fraud.

This new platform should have some level of eDiscovery built in, but not interfere with other tools you already own. Rather enable and support those Legal tools.

Finally, we think that data migration or data onboarding, should be built in.

Those would be the starting points of your new information governance platform.

The second challenge is your business also has some legal requirements. There are stakeholders, like a supervision team, and there might be different legal teams in different countries, with different legal requirements. With your legal requirements you might see if you can achieve everything with Microsoft Purview.

Unfortunately, there's a lot of problems in conjunction with Exchange Online: like the quality and performance of indexing - it doesn't do all the file formats and it can struggle with large files and so on.

Our customers come back and say: “We would love to do this all on the Microsoft platform, but for certain individuals we need something that is more reliable & more robust.”

To give you one real-life example: we have a number of customers who have done divestitures. These are sometimes a real problem because you want to filter the data that you are divesting. Let's say we want to stage everything for 500 people we want to include in the sell off to another company. We want to look for corporate IP, GDPR data, privacy protected data, health records and so on. We want to take that data out and then push this into the other tenant.

That’s pushing data at a scale where Purview and M365 would not be that useful.

The third challenge is, that most legal teams have their own tools. They might have NUIX or Relativity, and even if they have Purview, they might still feed the specialist legal tools. With this there is a question that all of these customers ask: Can you build an API to those tools, or can you at least give us PSTs in a certain format with a manifest file that we can read with those other tools?

They also want chain of custody for that extraction. So, how did we solve this? How did we meet all of these challenges?

Let’s take a look at Expireon.

Apart from being specialists at migrations, we looked at the challenges with M365, especially around legacy journal data. Similarly for voice or conferencing data. M365 is not really a great place to store your team's conference data and other things. If you look at Azure, you don't have the business logic to support searches. You can create an object store in Azure, where you just store everything but then how do I perform legal holds.

These are big issues.

We've built a platform which runs on docker. You can put it into Azure, or you can run it on AWS. It gives you the support for journal data and it has the business logic for running investigations. Our migration product is built in, which means you can take these additional sources and put them into Azure pretty seamlessly.

We bridge the gap between Azure and M365.

Overall, it's a pretty simple process.

You onboard the data.

Everything lives in Azure or in a cloud service of your choice.

As you see at the bottom of this diagram, you not only have storage management but also retention. Based on properties or content, you can retain or expire the data. From a legal and compliance perspective, there's an identification process, based on the users or custodians, or based on time frames. Then you can put this data into any of these other systems, on the right, in a completely integrated way.

Expireon operates within an ecosystem of tools, so it's a lightweight application but it communicates to all the favorite legal tools

No matter what data you have, whether it's data that is in mailbox format that is user-centric, up to the point where you have M&A remnants where you need to put them somewhere, we can put them into these two platforms.

Both live in the Microsoft cloud, and both are available to Microsoft Purview.

So, I hope you have a better understanding of the kind of data that is suitable for M365, what data causes issues, and how you can bring that into a modern, light-weight platform, to solve these issues while aligning closely with the Microsoft universe and the existing Legal ecosystem you already own.

And that’s everything for this presentation.

If you’d like to learn more visit cloudficient.com, and click on Expireon.