Organizing the committee’s requirements, led to the four key “criteria” categories, the first being service delivery. The service delivery centers around the ePortfolio platform attributes that correlate to providing and supporting the instance over time. Further collation of data resulted in six specific factors congealing under this main criterion: accountability for quality; support responsiveness and availability; lifecycle management, upgrades, and change management; user community; redundancy, and backup and recovery capability; and lastly, track record, and feedback from peers.
Accountability for quality is the solution provider’s ability to respond to the platform's shortcomings, fix the bugs, service outages, or feature requests. A provider at the top end of this metric will guarantee, in writing, a high degree of uptime, will provide terms of retribution for failure to comply with the contract, and be proactive and transparent in the tracking and remediation of both bugs and feature requests.
Support responsiveness and availability describes the need to have knowledgeable support personnel available 24/7/365. It also means that support requests are handled quickly, professionally, and that any higher order or more complex problems are escalated in a timely manner with appropriate and consistent feedback based on including prioritization and timelines for resolution wherever possible.
Lifecycle management upgrades, and change management describes a firm's maturity in providing a well-communicated upgrade and maintenance plan for their platform. A top contender in this category will have highly transparent and well documented testing and development practices, as well as detailed change logs which will consistently be published for clients.
Modern platform providers have found that by creating a user community around their products, self-help and user-driver content, can markedly enhance collaboration and coordination in product uptake. Any contender, strong in this factor will have a highly active, robust, public arena for the community of users to share and collaborate within.
Redundancy, backup, and recovery capabilities focus on a given platform’s ability to recover in case of minor or catastrophic failure. With regard to on premise solutions, this means the ability to restore functionality from backups in case of hardware failure, database corruption, or other unforeseen circumstances in a timely manner. For hosted solutions, this also includes the possibility of service fail-over to redundant hosting facilities in case of localized data center outages.
Track record and feedback from peers means that the company maintains a superb reputation for their product(s), their service, their responsiveness, and their product’s development and evolution over time. We find value in understanding and knowing if their customers are happy.
Costs are associated to the implementation range from highly specific to the more abstract. The choice will have specific functional cost requirements as well as ongoing, incidental costs, associated with the support and adoption of the platform. The cost factors defined are:
Initial cost or the buy-in cost. What are the initial costs for which will be borne on the university and to be expected to be spent up-front to purchase the platform. With software solutions, this is generally the largest single expenditure over the life of the product, though ongoing maintenance cost can easily summate to more over the life of the product stack.
Maintenance cost an ongoing, generally yearly, price you pay to maintain access to support, product updates, and for hosted solutions, the price to house and provide the infrastructure to deliver the platform to your audience.
Any complex system will require some amount of internal support cost. This is usually in the form of a subject matter expert or platform specialist; a technician or analyst in charge of monitoring, maintaining, and upgrading the system over its lifespan. These can range from a partial full-time equivalent employee (FTE) up to a whole team depending on the complexity, depth, and usage rate of a given system.
Licensing costs are another ongoing cost associated with many software and sometimes hardware products. In some cases, this is per-user, per-connection, per-processor. With some platforms, this is incorporated into the ongoing maintenance costs, in others, it is a specific, billable items, also typically collected yearly.
Training costs as described for this model refer to the startup training costs, required building expertise, and utility for a given platform across the appropriate user-base. This will include system administrator training (highly technical, small audience) to end-user training (less technical, larger audience). Although there may be ongoing costs associated to employee turn-over, the majority of this cost is borne up-front to get the incumbent staff up to speed.
Ongoing professional development costs are the more abstract, incidental costs associated with supporting a thriving platform. This includes travel and fees associated with staff development in the use and effectiveness of the platform. This would be similar to the training costs, but ascribed in an ongoing manner rather than the one-time cost of initial training.
The functional criterion is where the majority of the interaction with the platform choice occurs. This criterion describes the intended use, the primary features, and the robustness of its integration potential with other learning and online systems.
Privacy can be interpreted broadly but encompasses the user’s ability to control the availability of their learning artifacts. As some artifacts might be private or sensitive in nature, a robust solution will provide artifact-level permissions and sharing controls so that students and educators can choose what is revealed precisely to what audience.
Customization and personalization capabilities describe the ability to transform the platform in two ways; for the user, and for the institution. The user will want to create a place that reflects their personality and experience as this will drive adoption and engagement. The institution needs to be able to customize the interface with branding, and templates for programs or courses.
Students are increasingly using social media platforms as an extension of their physical lives; to allow the extension of their academic lives into this digital realm would be a powerful tool for engagement and potentially lead to all kinds of new ways of learning. Integration that crosses various social media and rich media platforms will bridge the gap between an academic-only system and the public, life-long portfolio experience.
While creating and displaying content is the initial, primary goal, a system that provides workflow capabilities such that automation mechanisms can be developed around learning modules or assessment would greatly improve adoption and potentially be a huge time-saver. One example would be to automate the random selection of portfolios for assessment and an assessment process that simplifies and reduces the time to analyze each portfolio.
Assessment, analysis, and reporting is a big category describing a big set of tools. Assessment is the secondary goal of the project and the system should allow for assessment of any arbitrary collection of artifacts that have been presented within the ePortfolio environment. Analysis adds a layer of cataloging and reporting provides a means to assimilate that analysis into useful, meaningful, and repeatable records.
Accessibility is key attribute for today’s online systems as they must be flexible enough to interface with various technologies designed to provide consistent access to technology regardless of physical or mental disability or impairment.
As part of the primary goal of learning, pedagogical utility is the platforms' ability to deliver on this promise. Are their elements that present a roadblock to learning and engagement? What design elements are novel, useful, and approachable that will assist in this end?
The final criterion category encompasses the technical requirements for a large, university-wide system such as this. The factors involved are
Identity, data, and access management which has to do with properly identifying users, getting them logged in, and ensuring that only they have access to that which they are provisioned. This category starts with supporting single sign-on so that the existing account infrastructure and lifecycle can be leveraged. It also includes tools and techniques built into the system to manage and realize the types of permission granularity described under privacy.
System and application administration and management capabilities include the satellite of tools used for administering and managing the environment. A strong system will have thought of the flexible needs and nature of such a system and designed their tools to minimize the amount of work necessary to implement both systematic and novel changes, both on the small and large scale.
Security is the systematic thoughtful implementation of security into the design and implementation of the product. Larger solutions generally provide a white paper describing both the development methodology and service delivery design consideration and mechanisms to create, maintain, monitor, and respond to security concerns. Any viable product offering will provide this type of robust, transparent communication regarding the security infrastructure.
Separate from the functional integration is the technical integration. Whereas, functional focused on the uses, this integration factor refers to integration with existing tools and services owned by the university. This includes our student information system, our LMS, Google Apps for Education, and others. A successful platform should have vehicles for tying these together with minimal effort on the part of our organization.
Interoperability is becoming ubiquitous in the modern era, but it is not necessarily common or robust when it comes to larger platform providers. A winning solution should clearly communicate their capabilities across a diverse set of user-devices (mobile devices, tablets, browsers) and be as agnostic as possible. The provider should also show a track record of adoption as new technologies arise as this will correlate to the nimbleness and adaptability of both the provider and the platform.
For this project, we had direct access to several of the key-players involved in the ePortfolio platform process. From this population, we formed five distinct panels for conducting our model validation and pairwise comparisons: criterion level, service delivery factors, cost factors, functional factors, and technical factors.
The interview process was fruitful, as each panelist provided valuable feedback for our model development, and each of the individual’s backgrounds correlated extremely well within the model. This is reflected further in the analysis section as the consistency and disagreement values were all extremely low.