What are the guiding considerations for EaaSI development?
EaaSI is a management system for emulated computer environments, software, and content, enabling users to create, discover, administer, and share these resources for access and reuse. Our development priorities are informed by the overall goal of the EaaSI program of work:
To scale up access to and use of emulation and software preservation functionality
For us this means doing everything we can to increase the number of people that are using EaaSI on a regular basis. We do this by introducing new tools and services (and improving existing ones) to make engagement with emulation and legacy software more approachable.
Providing and Maintaining Core Tools and Services to Increase Access to Emulation and Software Preservation Functionality
The Base – Emulation-as-a-Service
Emulation-as-a-Service (EaaS), the core technology underlying EaaSI, solves the problem of having sufficient expertise to find, configure, and use emulators to run legacy software. It does so by making accessible pre-configured emulators via just a web-browser.
Using EaaS users can run emulators to create virtual computers with software installation media attached and install software on the emulated computers before saving the result as an “environment”. The configured computers with software installed on them are what we call “environments”.
EaaS also significantly reduces storage costs when using emulation at scale. It does this by allowing users to “derive” environments from others. This means that a user can take an existing 4GB environment, install a new 600MB application, and save the changed environment as a “derivative” environment that only requires storing the additional 600MB added by installing the application. Previously, if a user had wanted to save the two separate environments it would have required storing 4GB (original environment) + 4GB (copy of the original) + 600MB (data from the installed application), i.e. 8.6GB of data. In this example, the user saves 4GB of data (47%) that they would otherwise have to store.
Using EaaS as the base of EaaSI we can significantly simplify the user experience of setting up and running configured environments in emulators. EaaS provides the tools so you can quickly and easily create your emulated computer environment.
EaaSI enables users to share software installation media and pre-configured emulated computer environments between members of “EaaSI networks”. This is transformative, as it means users now rarely have to find, acquire, configure, and describe software themselves. They can simply look for the software in an EaaSI network and replicate it for use in their local installation with just a few clicks in their web browser. The time and resource savings here are immense and will grow even more quickly as more organizations participate in EaaSI networks (currently there is one primary EaaSI network but there is the potential for a federated series of networks across the globe) and add more assets to them.
Description and Discovery
Making software and environments discoverable and shareable in EaaSI requires a significant amount of work. Without cataloguing and discovery infrastructure for software and environments, users can share assets between network nodes but aren’t easily able to discover, add, or describe new assets. During the first phase of EaaSI, the team developed a basic cataloguing and discovery system for software, software installation media, and configured computing environments that must be expanded in future releases to accommodate a wealth of descriptive, administrative, and technical information that will be captured by the system. While there have been similar projects in the past, none have had the scope of what the team is building in EaaSI. This has made the development process particularly challenging as we have had to develop novel approaches for collecting, structuring, and sharing metadata.
The same challenging situation applies to the description and management of emulators. There are plenty of open source emulators available covering most relevant computer platforms. Within previous EaaS work and the ongoing EaaSI program of work, a major task is to describe and package emulators. While previous work focused on technical framework integration and usability, our focus is now on discoverability, managing and maintaining essential emulators for the preservation community, ensuring consistent (reproducible) runtime results and maintain different versions.
This functionality is still in a nascent form. There is still a great deal of work to do to improve the cataloguing, browsing, and searching features in EaaSI and to improve the sharing and reuse of metadata captured in EaaSI. However, it already adds great value and makes working with legacy software and emulation much less daunting for new users.
The Universal Virtual Interactor
A tool for automating using computer environments in EaaSI to interact with digital objects, the UVI includes provides functionality that matches digital objects to software applications and environments in EaaSI (using extensive metadata that is being integrated with Wikidata as much as possible). This can be used during appraisal, accessioning, ingest, and at point of access to identify software dependencies of digital objects, quantify the risk of loss of access to the objects, and with the other UVI functions, provide direct access to the objects in appropriate legacy software. For example the UVI functions can be embedded in digital access platforms to enable one-click in-browser access to born-digital content in software contemporaneous to the objects. In time we are working to increase the applications of the UVI to include additional abilities like the automatic running of Docker containers in external workflows, and automatic access to legacy databases and web servers.
Easy access to emulated environments
End user support is central to our effort to increase use and access of emulation and we see emulation as a powerful tool for expanding accessibility to the growing collections of digital material within cultural heritage organizations. To that end, we have planned from the start of the EaaSI program to develop virtual reading room features for secure access to emulation environments for research and study. This includes the implementation of core infrastructure like access control lists and improved security management of backend storage. We want to ensure that whether you’re providing broad access to circulating library materials or require strict limitations for special collections, EaaSI can provide the necessary settings (e.g., IP restrictions, user authentication, etc.) to meet your needs.
Reducing Barriers to the Adoption of Emulation at Scale
We undertake many other activities that are not directly related to software development such as support and troubleshooting, adding content to the network, outreach, communications and community management, planning, budget management, general administration, policy development, legal clarification, fundraising and more. So while software development is a large part of our work, our small team is also busy with many other tasks which inevitably limit our development capacity. Each of these requires time from our development teams and each is weighed against each other when we prioritise their work.
Maintenance is especially important to a long term effort like EaaSI as we have a strong motivation and obligation to ensure that we can continue to use EaaSI over the very long term. So alongside feature development you will see plenty of maintenance activities included in our development roadmap. Over time we also need to maintain integrated emulators and upgrade to current versions so that we can continue to ensure access to the software environments that the emulated hardware supports, but also
improve integration to cope with new use cases and usage scenarios.
Making EaaSI Affordable
To lower costs we also need to make the software more efficient and cheaper to run and, importantly, cheaper to maintain. Lowering maintenance costs is difficult, but it also relies on regular maintenance being undertaken. A product maintained irregularly will be more expensive to maintain over time than one that is maintained regularly.
Lowering actual costs (that our users will have to contribute to) can also be achieved via economies of scale. For us, this represents the ability to reduce per-organization costs by increasing the total number of organizations we support. Furthermore, to be self-sustaining at all, we need sufficient ongoing income to cover our costs. Both of these factors mean that we will sometimes focus on developing aspects of the EaaSI software suite that support increasing the number of paying users. For example, we may spend time developing code that supports marketing activities, like expanding the features in the freely available open source portal in order to improve the likelihood of the portal attracting new users to the fee-supported services we will offer.
We also need to spend time ensuring that we can grow the technology as we add more users. Additional users means more resources required to efficiently operate emulation sessions, manage access to software and environments, and maintain consistent availability of the system. We must also keep pace with technological advancement and anticipate comprehensive overhauls of the system over the years to keep it up-to-date with new standards for software design and architecture.
Making EaaSI Integral Infrastructure
Common conceptions of “infrastructure” include technologies that are fundamental to a functioning society such as roads, power stations, or water mains; the services they provide such as waste disposal, public transportation, or emergency response; and the people that enable all of this. This traditional infrastructure enables other parts of society to function and survive. It is very difficult to imagine modern society without this infrastructure and we are working to position EaaSI in a similar place. We aim to embed EaaSI as a fundamental/core feature set that other tools and services are built on top of; to position EaaSI as a feature set that is assumed to always be available. To achieve this goal, we are working to ensure EaaSI has extensive, clear, documentation and robust APIs to support interoperability and the integration of EaaSI into third party workflows and as a backend to third-party tools and services. A lot of this work is somewhat “invisible” but it is vital to achieving this goal of EaaSI as infrastructure.
An example of these efforts is our ongoing collaboration with the Siegfried project. We have been working with the Siegfried team to enable their service to use metadata from Wikidata to power the file format matching it currently performs. This includes metadata describing tens of thousands of software titles and thousands of file formats that the EaaSI team has added to Wikidata over the course of the program of work. We are aiming to eventually enable Siegfried to be able to do some of the software matching that the UVI tooling currently enables. This would allow anyone already using Siegfried to use it scan their objects and see what environments in EaaSI they depend on. This in turn would help to highlight the necessity of EaaSI for ensuring long term access to their digital content.
Yale and EaaSI continue to contribute to the development of Wikidp.org, a curation portal for digital preservationists to add data to Wikidata. Yale and EaaSI support ongoing work to create and contribute substantial metadata about software and file formats to Wikidata.
- Creating and supporting accessible demonstrations of the value of EaaSI to help with making the case for implementing it
- We work closely with users to provide support when they want to use EaaSI in demonstrations
- We are planning additional development on the publicly accessible open source portal to enable anyone to try the full EaaSI feature set using open source software examples. This will be able to be used by anyone wanting to demonstrate the value of EaaSI in their local organizations.
Supporting EaaSI and Its Users
We strive to provide comprehensive documentation, via the EaaSI User Handbook and the Community Forum, to reduce unknowns and help instill confidence should an organization decide to implement EaaSI. Upkeep of these resources requires regular input from our development team which has to be balanced against our other priorities but is vital to achieving this goal.
We also provide responsive and meaningful support, primarily to help users, but also to reduce any perceived risk of implementing EaaSI. Our developers spend a great deal of time on user support, reducing the time they have available for other activities. However we recognize the value of this work and try to be as responsive as possible. We hope this approach makes EaaSI less risky for organizations by providing the reassurance that there will be help available quickly if anything goes wrong.
What are our development commitments?
EaaSI is currently funded by two generous organizations, the Alfred P. Sloan Foundation, and the Andrew W. Mellon Foundation. As part of our two proposals to the foundations (we’re in our second phase of funded work right now) we promised to develop many new features on top of EaaS. A portion of these features are complete but we have many remaining to develop and implement. A full list of development deliverables promised to our funders follows.
- Automatable interaction with emulated software environments – These services will enable users to interact with and record interactions with software environments. This functionality is in the early stages of research and development, but we foresee these services as a core element of EaaSI workflows in the future, further enabling the automation of computational tasks for installation of software, migration of file formats, and rendering files.
- Mobile device emulation – OpenSLX will add support for the core open source Android emulator to enable the preservation and reuse of mobile apps and data created and used on Android devices.
- Networked environments – Emulation of software environments connected via emulated networks will be incorporated into the EaaSI system with corresponding updates to the data model and visual elements to support discovery, navigation, and interaction of the networks of software environments.
- EaaSI public sandbox – We maintain an instance of EaaSI that is publicly available and contains only open source software. In its current state, environments in the sandbox can only be accessed for demonstration purposes. In the future, the sandbox will remain limited in scope to open source software resources, but users will be able to configure and save new environments and view their own files using configured software. EaaSI nodes will benefit from the participation of public users as new open source environments are created and documented by volunteers and subsequently made available in the EaaSI network.
- Universal Virtual Interactor (UVI) – We have a prototype of the Universal Virtual Interactor and are working to make it production-ready. As described above, the UVI matches file formats with configured software in emulation environments and automates the rendering of files in compatible software. This is essential infrastructure for the EaaSI system, as it will allow users to gain access to their files without having to decipher which software is necessary to do so.
- Virtual reading rooms – We’re working to enable EaaSI users to configure virtual desktops that can be configured with secure content and provided to researchers for periods of time and with various additional access restrictions in place.
- Reference photographs – Inclusion of reference photographs throughout the EaaSI interface will help users make the logical connection between a digital object and its physical analog. Incorporation of environment screenshots (e.g. an image of the user-interface of each configured application) in the EaaSI interface will aid users when browsing through the many available resources in a node and the network.
Our development decision-making
A development planning meeting takes place twice monthly to guide the day-to-day implementation of the roadmap and make sure that everyone involved in Strategic Planning and User Support is up-to-date on the current status of development (feature status, proposed focus for the next two weeks, immediate needs/concerns, etc.)
The development planning meetings balance input from:
- Back-end/core EaaS development (OpenSLX)
- UI/UX development (OpenSLX and sub-contracted Dual Lab)
- Quality control and bug reports (OpenSLX+ User Support Lead)
- Community feedback (User Support Lead)
- Service sustainability (Strategic Planning/governance)
- Committed deliverables (Strategic Planning/governance)
Any technical decisions with implications for staffing capacity, financial resources, the development roadmap or the direction of the EaaSI program of work as a whole are raised to the Strategic Planning group (Seth Anderson, Euan Cochrane, Klaus Rechert), who also meet every two weeks or as-needed to resolve such strategic questions.
How will you know what is being developed when?
As of August 19th 2021, we are regularly updating the EaaSI forum with information on our current development activities. These will be posted monthly to the Community Forum following every other development planning meeting, and outline the work we aim to complete and deliver in the following month.
The development roadmap is also posted to the Community Forum and will receive regular updates as needed.