Automated assessment of secure search systems
January 20, 2015
ACM SIGOPS Operating Systems Review, Vol. 49, No. 1, 20 January 2015, pp. 22-30.
This work presents the results of a three-year project that assessed nine different privacy-preserving data search systems. We detail the design of a software assessment framework that focuses on low system footprint, repeatability, and reusability. A unique achievement of this project was the automation and integration of the entire test process, from the production and execution of tests to the generation of human-readable evaluation reports. We synthesize our experiences into a set of simple mantras that we recommend following in the design of any assessment framework.