--- title: How to Move a 120 TB Web Archive to the Cloud in Two Weeks description: Learn all about how we moved a 120 TB website archive to the cloud for The National Archives. --- [Skip to content](https://www.mirrorweb.com/blog/how-to-move-120-tb-web-archive-to-the-cloud#main-content) [![Master Logo - ColourWhite Landscape](https://www.mirrorweb.com/hs-fs/hubfs/Master%20Logo%20-%20Colour%EF%80%A2White%20Landscape.png?width=178&height=40&name=Master%20Logo%20-%20Colour%EF%80%A2White%20Landscape.png "Master Logo - ColourWhite Landscape")](https://www.mirrorweb.com/) - [Unified Platform](https://www.mirrorweb.com/platform/) Show submenu for Unified Platform - [Mira - Supervision Agent](https://www.mirrorweb.com/mira) - [Website Archiving](https://www.mirrorweb.com/platform/website-archiving/) - [Email & Communication Archiving](https://www.mirrorweb.com/platform/email-archiving/) - [Social Media Archiving](https://www.mirrorweb.com/platform/social-media-archiving/) - [Mobile Archiving](https://www.mirrorweb.com/platform/mobile-archiving/) - Solutions Show submenu for Solutions - [By Industry](https://www.mirrorweb.com/industries/) Show submenu for By Industry - [Financial Services](https://www.mirrorweb.com/industries/financial-services/) - [Government](https://www.mirrorweb.com/industries/government/) - [Healthcare](https://www.mirrorweb.com/industries/healthcare-pharma/) - [Insurance](https://www.mirrorweb.com/industries/insurance/) - [Legal](https://www.mirrorweb.com/industries/legal/) - [Technology](https://www.mirrorweb.com/industries/technology/) - [By Use Case](https://www.mirrorweb.com/use-cases/) Show submenu for By Use Case - [Agentic Communications Supervision](https://www.mirrorweb.com/mira) - [Communication Capture](https://www.mirrorweb.com/use-cases/communication-capture/) - [Prevent Compliance Violations](https://www.mirrorweb.com/use-cases/prevent-compliance-violations) - [Audit-Ready Recordkeeping](https://www.mirrorweb.com/use-cases/audit-ready-automated-record-keeping/) - [Reduce Costs & Improve Efficiency](https://www.mirrorweb.com/use-cases/compliance-cost-efficiency/) - [Website Archiving](https://www.mirrorweb.com/use-cases/website-archiving/) - [By Channel](https://www.mirrorweb.com/channels/) Show submenu for By Channel - [Website](https://www.mirrorweb.com/use-cases/website-archiving/) - [Email](https://www.mirrorweb.com/channels/email/) - [Social Media](https://www.mirrorweb.com/channels/social-media/) - [Instant Messaging](https://www.mirrorweb.com/channels/instant-messaging/) - [Mobile](https://www.mirrorweb.com/channels/mobile/) - Why MirrorWeb Show submenu for Why MirrorWeb - [Customer Success](https://www.mirrorweb.com/case-studies) - [About Us](https://www.mirrorweb.com/why-mirrorweb/about) - [Careers](https://www.mirrorweb.com/careers/) - [Partnerships](https://www.mirrorweb.com/partnerships/) - Resources Show submenu for Resources - [Blog](https://www.mirrorweb.com/blog) - [Events](https://www.mirrorweb.com/events) - [Help Center](https://help.mirrorweb.com/en/) - [Legal Resources](https://www.mirrorweb.com/help-centre/legal/) [Request a demo](https://www.mirrorweb.com/demo) [Login](https://my.mirrorweb.com/login) # How to Move a 120 TB Web Archive to the Cloud in Two Weeks [![](https://www.mirrorweb.com/hubfs/Blog_Author.png) MirrorWeb](https://www.mirrorweb.com/blog/author/mirrorweb) [Historic Content](https://www.mirrorweb.com/blog/tag/historic-content) Published: 19 Jun 2018 ![blog placeholder](https://www.mirrorweb.com/hubfs/raw_assets/public/mirrorweb-theme/images/blog-placeholder.jpg) #### Table of Contents [Contact Us](https://www.mirrorweb.com/contact/) Share Building a resource as big as the [UK Government Web Archive](https://www.mirrorweb.com/the-national-archives-case-study) takes a lot of storage space. Comprising over 5,000 websites from 1996 to the present, as well as tweets and videos from government social media accounts, the data footprint of the archive in 2018 is over 120TB ‚Äì far bigger than the average consumer hard-drive size of between 500GB and 1TB. This creates challenges for the engineers who need to manage and manipulate that data ‚Äì something MirrorWeb discovered in 2017, when we were awarded the contract to relaunch the archive as a more modern, cloud-based service. In order to address these challenges and implement the version of the web and social archive you see today, there were a couple of places we needed to think outside the box and even design our own tools and workflows to make the process as fast and effective as possible.    ## Why archive in the cloud? First, though: why would MirrorWeb and The National Archives go to the trouble of moving the UK Government Web Archive to the cloud in the first place? One of the reasons is the sheer volume of data itself. As an archive grows, it becomes less and less sustainable for an organisation like The National Archives ‚Äì or its archiving partner ‚Äì to keep investing in new infrastructure to accommodate this growth. However, when you source virtual infrastructure from the likes of Amazon Web Services (AWS) ‚Äì the cloud platform used by massive web brands such as Netflix and Airbnb ‚Äì it becomes trivial to add new storage whenever you need it. (That‚Äôs not to say cloud providers don‚Äôt rely on physical hardware ‚Äì they simply have the economies of scale to offer almost unlimited capacity.) Cloud can also make web-based services faster and more reliable. Put in the simplest terms, physical hardware ‚Äì like servers and hard drives ‚Äì can be overloaded or fail. Cloud infrastructure, on the other-hand, tends to have a higher level of redundancy built-in ‚Äì so if there‚Äôs ever a problem with a hard drive, server or even data centre, your services can simply be resumed elsewhere with minimal disruption. ***Find out more about website archiving from MirrorWeb in our free eBook.*** ## Our first challenge: moving the data However, much as we knew the UK Government Web Archive would benefit from moving into the cloud, we still needed to solve the problem of getting it there in the first place. For one thing, transferring 120TB of data from one place to another across a network can take a long time. What‚Äôs more, a transfer on this scale can be expensive (due to network costs) and even a potential security risk without the right security controls in place. At that time, the archive was stored in a data centre in Paris (so it wasn‚Äôt difficult for us to get our hands on the data itself) across 72 USB-3 hard drives. We decided our best option for the transfer was to use devices called AWS Snowballs, which connect to your local network, copy and encrypt your data to internal hard drives, and can then be shipped to an AWS data centre for transfer into the cloud. To speed things up, we also brought along two of our own custom-built PCs that would allow us to move data from up to sixteen of the USB-3 hard drives at a time. With this ensemble of 72 hard drives, two custom-built PCs and two AWS Snowballs, the entire process took just two weeks ‚Äì not bad for more than two decades of internet history! ## Our second challenge: indexing the data Of course, moving a national archive to the cloud is a waste of time if the data isn‚Äôt accessible to the researchers, students and members of the public who need to use it. This meant our next job was to build a public-facing website where visitors could view websites and social media content in the website in their original form, as well as search for content on specific topics. The latter was a particular challenge for us. Search may seem like one of the most basic web technologies, but it can be complex to implement. This is because search engines don‚Äôt scan an entire set of documents one by one ‚Äì they use indexes, which helps them return useful results much faster. MirrorWeb was tasked with writing a complete replacement for the UK Government Web Archives‚Äô previous search functionality, which meant we needed to index 1.4 billion documents from scratch. Once again, we ended up using a combination of our own and off-the-shelf technologies to do this. We struggled to find an existing tool that would meet our specific need for indexing a large number of small files, so we built something called WarpPipe. This allowed us to index all The National Archives‚Äô documents in just ten hours ‚Äì far below the timeframe of six to eight weeks we were told it would take with one of the most popular big data processing tools. The search functionality itself is provided by Elasticsearch, which substantially improves on The National Archives‚Äô previous search engine in terms of speed, flexibility and reliability. We now update the index once a month rather than once a quarter, for example, so it‚Äôs much faster for new archive content to show up in search results. ## The outcome: so what are the implications of this work for the archive‚Äôs users? One benefit is that in moving to the cloud, we cut down on a lot of the overhead that comes with looking after your own infrastructure. We don‚Äôt need to work hard just to keep the lights on, which means the National Archives and MirrorWeb can focus on adding to and improving the features of the archive that our users care about. These range from simple user interface improvements to more advanced capabilities, like transferring large amounts of data for large-scale research projects. It also opens doors to a lot of new opportunities for how we develop the UK Government Web Archive in the future. With all our content now stored in a modern, flexible cloud environment, it‚Äôs become much simpler to process this data ‚Äì as we did with WarpPipe ‚Äì and present it in a way that helps our users unearth and understand more about our shared digital heritage. [Curious to see what we did? You can take a look at the live public archive right here](http://www.nationalarchives.gov.uk/webarchive/) If you would like to see how state-of-the-art cloud-based archiving can benefit your organisation, download our free eBook '**A Guide to Website Archiving**' today. ## Read On ![blog placeholder](https://www.mirrorweb.com/hubfs/raw_assets/public/mirrorweb-theme/images/blog-placeholder.jpg) #### [5 simple ways to get the most out of your web archives](https://www.mirrorweb.com/blog/five-ways-to-get-the-most-out-of-web-archives) [Read story](https://www.mirrorweb.com/blog/five-ways-to-get-the-most-out-of-web-archives) ![blog placeholder](https://www.mirrorweb.com/hubfs/raw_assets/public/mirrorweb-theme/images/blog-placeholder.jpg) #### [AWS re:Invent 2017 - The UK Government Web Archive Project](https://www.mirrorweb.com/blog/uk-government-web-archive-mirrorweb) In 2017 we were invited to speak at the AWS re:Invent event in Las Vegas about our web archiving... [Read story](https://www.mirrorweb.com/blog/uk-government-web-archive-mirrorweb) ![blog placeholder](https://www.mirrorweb.com/hubfs/raw_assets/public/mirrorweb-theme/images/blog-placeholder.jpg) #### [The social media archive is now fully searchable!](https://www.mirrorweb.com/blog/the-national-archives-relaunch-the-social-media-archive) Our social media archive is now the first of its kind in the UK to feature a search function,... [Read story](https://www.mirrorweb.com/blog/the-national-archives-relaunch-the-social-media-archive) [![Master Logo - ColourWhite Landscape](https://www.mirrorweb.com/hs-fs/hubfs/Master%20Logo%20-%20Colour%EF%80%A2White%20Landscape.png?width=178&height=40&name=Master%20Logo%20-%20Colour%EF%80%A2White%20Landscape.png "Master Logo - ColourWhite Landscape")](https://www.mirrorweb.com/) Stay Connected. Stay Compliant. #### Quick Links - [Communication Capture](https://www.mirrorweb.com/use-cases/communication-capture/) - [Prevent Compliance Violations](https://www.mirrorweb.com/use-cases/prevent-compliance-violations) - [Audit-Ready Recordkeeping](https://www.mirrorweb.com/use-cases/audit-ready-automated-record-keeping/) - [Supervising Digital Communications](https://www.mirrorweb.com/use-cases/supervised-digital-communication/) - [Reduce Costs & Improve Efficiency](https://www.mirrorweb.com/use-cases/compliance-cost-efficiency/) #### Find Us **North America** 701 Brazos St Suite 970 Austin, Texas 78701-3232 **United Kingdom** Manchester House 18-22 Bridge Street Manchester M3 3BZ #### Speak to us If you have any questions, please don’t hesitate to get in touch! [Contact Us](https://www.mirrorweb.com/contact/) © MirrorWeb 2026 • [Terms](https://www.mirrorweb.com/help-centre/legal/terms/) | [Privacy Policy](https://www.mirrorweb.com/help-centre/legal/privacy-policy/) ![673cba813eccf634bab46632_Motherboard 300 x 300 signatory badge](https://www.mirrorweb.com/hs-fs/hubfs/673cba813eccf634bab46632_Motherboard%20300%20x%20300%20signatory%20badge.jpg?width=200&height=200&name=673cba813eccf634bab46632_Motherboard%20300%20x%20300%20signatory%20badge.jpg) ![65eeeb5ba22a2ed7014d4f4d_21972-312_SOC_NonCPA](https://www.mirrorweb.com/hs-fs/hubfs/65eeeb5ba22a2ed7014d4f4d_21972-312_SOC_NonCPA.png?width=200&height=200&name=65eeeb5ba22a2ed7014d4f4d_21972-312_SOC_NonCPA.png) ![64b658f3e0cbdcbe04e83981_27001 RGB Purple-p-500](https://www.mirrorweb.com/hs-fs/hubfs/64b658f3e0cbdcbe04e83981_27001%20RGB%20Purple-p-500.png?width=200&height=200&name=64b658f3e0cbdcbe04e83981_27001%20RGB%20Purple-p-500.png) ![64b6590db2301cbb306c2e34_9001 RGB Purple-p-500](https://www.mirrorweb.com/hs-fs/hubfs/64b6590db2301cbb306c2e34_9001%20RGB%20Purple-p-500.png?width=200&height=200&name=64b6590db2301cbb306c2e34_9001%20RGB%20Purple-p-500.png) ```json { "@context" : "https://schema.org", "@type" : "BlogPosting", "author" : { "@type" : "Person", "name" : "MirrorWeb", "url" : "https://www.mirrorweb.com/blog/author/mirrorweb" }, "dateModified" : "2025-04-24T08:18:46.563Z", "datePublished" : "2018-06-19T05:00:00.000Z", "headline" : "How to Move a 120 TB Web Archive to the Cloud in Two Weeks", "mainEntityOfPage" : { "@id" : "https://www.mirrorweb.com/blog/how-to-move-120-tb-web-archive-to-the-cloud", "@type" : "WebPage" }, "publisher" : { "@type" : "Organization", "logo" : { "@type" : "ImageObject", "url" : "https://www.mirrorweb.com/hubfs/Offical%20Logos%20-%20Flat%20Colours%202024_Black%20Logo-1.png" }, "name" : "MirrorWeb, Inc." } } ``` ```json { "@context" : "https://schema.org", "@id" : "https://www.mirrorweb.com/#organization", "@type" : "Organization", "areaServed" : [ { "@type" : "Country", "identifier" : "US", "name" : "United States" }, { "@type" : "Country", "identifier" : "GB", "name" : "United Kingdom" } ], "contactPoint" : { "@type" : "ContactPoint", "areaServed" : [ "US", "GB" ], "contactType" : "sales", "url" : "https://www.mirrorweb.com/demo" }, "description" : "MirrorWeb is the communications supervision platform for SEC- and FINRA-regulated financial services firms, unifying communications capture, archiving, and AI-native supervision.", "foundingDate" : "2012", "foundingLocation" : { "@type" : "Place", "name" : "Manchester, United Kingdom" }, "knowsAbout" : [ "communications supervision", "AI communications supervision", "electronic communications archiving", "email archiving", "SEC Rule 17a-4 compliance", "WORM storage", "FINRA Rule 3110 compliance", "Written Supervisory Procedures", "off-channel communications compliance", "BYOD archiving", "WhatsApp compliance", "mobile communications archiving", "social media archiving", "website archiving", "web content archiving", "FINRA Rule 2210 compliance", "SEC marketing rule compliance", "web changes tracking", "eDiscovery", "digital communications governance and archiving", "communications compliance software", "RegTech" ], "knowsLanguage" : [ "en" ], "legalName" : "MirrorWeb Inc.", "logo" : { "@id" : "https://www.mirrorweb.com/#logo", "@type" : "ImageObject", "url" : "https://www.mirrorweb.com/hubfs/master-logo.png" }, "name" : "MirrorWeb", "sameAs" : [ "https://www.linkedin.com/company/mirrorweb", "https://www.youtube.com/@mirrorweb", "https://www.g2.com/products/mirrorweb", "https://www.capterra.com/p/171780/Website-Social-Media-Archiving/", "https://www.crunchbase.com/organization/mirrorweb" ], "slogan" : "The communications supervision platform.", "url" : "https://www.mirrorweb.com/" } ``` ```json { "@context" : "https://schema.org", "@id" : "https://www.mirrorweb.com/#website", "@type" : "WebSite", "name" : "MirrorWeb", "publisher" : { "@id" : "https://www.mirrorweb.com/#organization" }, "url" : "https://www.mirrorweb.com/" } ```