We remember the things that are important to us. We capture, store and make our memories accessible so that we can learn from them in the future. Archiving does the same thing. It enables us to store, index and return to documents and records of historical and cultural significance.
The more memories, content, data that we create, the more heightened the need is for an organised, accessible and effective memory bank.
The digital world has vastly accelerated this need.
Millions upon millions of pages of web content are published every day, and the average page has a lifespan of just 90 days before it is replaced. Social media platforms contain significant amounts of personal and business data and more is created every second.
That's why we've developed the world's most advanced digital public archiving solution. With our intuitive search portal, you can find web pages, social media posts, images and documents in a straightforward, useful way. Without a secure but accessible public portal digital communications are at risk of being hidden or even lost forever.
We are here to make sure the important things never get forgotten.
We were approached by The National Archives to build a public digital archiving solution for the UK Government Web Archive (UKGWA).
We needed a way for the public to quickly and easily search through some 120TB of data which we had captured and stored for the UKGWA. It had to enable replay of all archives, as well as full-text search, all in a managable solution.
We decided the best option was to build an indexing and search solution from scratch, where we could provide both video and image replay functionality as part of the service. And the tool building didn't stop there. Traditional data software such as Hadoop were not suitable, so we created our own - it's called WarpPipe, a specific data tool for the cloud.