Publications

Precise estimation of uncertainty in predictions for AI systems is a critical factor in ensuring trust and safety. Deep neural networks trained with a conventional method are prone to over-confident predictions. In contrast to Bayesian neural networks that learn approximate distributions on weights to infer prediction confidence, we propose a novel method, Information Aware Dirichlet networks, that learn an explicit Dirichlet prior distribution on predictive distributions by minimizing a bound on the expected max norm of the prediction error and penalizing information associated with incorrect outcomes. Properties of the new cost function are derived to indicate how improved uncertainty estimation is achieved. Experiments using real datasets show that our technique outperforms, by a large margin, state-of-the-art neural networks for estimating within-distribution and out-of-distribution uncertainty, and detecting adversarial examples.

The field of human activity recognition (HAR) often utilizes wearable sensors and machine learning techniques in order to identify the actions of the subject. This paper considers the activity recognition of walking and running while using a support vector machine (SVM) that was trained on principal components derived from wearable sensor data. An ablation analysis is performed in order to select the subset of sensors that yield the highest classification accuracy. The paper also compares principal components across trials to inform the similarity of the trials. Five subjects were instructed to perform standing, walking, running, and sprinting on a self-paced treadmill, and the data were recorded while using surface electromyography sensors (sEMGs), inertial measurement units (IMUs), and force plates. When all of the sensors were included, the SVM had over 90% classification accuracy using only the first three principal components of the data with the classes of stand, walk, and run/sprint (combined run and sprint class). It was found that sensors that were placed only on the lower leg produce higher accuracies than sensors placed on the upper leg. There was a small decrease in accuracy when the force plates are ablated, but the difference may not be operationally relevant. Using only accelerometers without sEMGs was shown to decrease the accuracy of the SVM.

RTCA has been developing Minimum Operational Performance Standards (MOPS) for Detect and Avoid (DAA) and Command and Control (C2) systems as part of Special Committee – 228 (SC-228). The Phase 1 MOPS were published in 2017 and a Phase 2 effort to revise and extend the Phase 1 MOPS is ongoing. In order for the MOPS to be fully utilized, they must be evaluated by the FAA employing the FAA's Safety Risk Management (SRM) process. In order to support the SRM process, there is a need for simulation data focused on the safety of DAA encounters. This analysis focuses on gathering information to validate the use of SC-228 MOPS compliant DAA and C2 systems to enable routine UAS operations in the National Airspace System (NAS) without a chase aircraft or visual observers. The scope of this effort aligns with the SC-228 Terms of Reference (TOR) that can be generally characterized as UAS flying IFR and receiving ATC separation services. This analysis evaluates the system safety in mixed classes B, C, D, E, and G airspaces, and includes IFR, VFR, Cooperative, and Non-Cooperative aircraft. This analysis plan describes the four analysis tasks (Section 2) and the simulation plan (Section 3) that will be executed to accomplish these tasks. It is expected that this analysis plan will be extended to include the analysis results and become the final deliverable.

Microwave atomic clocks have traditionally served as the 'gold standard' for precision measurements of time and frequency. However, over the past decade, optical atomic clocks have surpassed the precision of their microwave counterparts by two orders of magnitude or more. Extant optical clocks occupy volumes of more than one cubic metre, and it is a substantial challenge to enable these clocks to operate in field environments, which requires the ruggedization and miniaturization of the atomic reference and clock laser along with their supporting lasers and electronics. In terms of the clock laser, prior laboratory demonstrations of optical clocks have relied on the exceptional performance gained through stabilization using bulk cavities, which unfortunately necessitates the use of vacuum and also renders the laser susceptible to vibration-induced noise. Here, using a stimulated Brillouin scattering laser subsystem that has a reduced cavity volume and operates without vacuum, we demonstrate a promising component of a portable optical atomic clock architecture. We interrogate a 88Sr+ ion with our stimulated Brillouin scattering laser and achieve a clock exhibiting short-term stability of 3.9 × 10−14 over one second—an improvement of an order of magnitude over state-of-the-art microwave clocks. This performance increase within a potentially portable system presents a compelling avenue for substantially improving existing technology, such as the global positioning system, and also for enabling the exploration of topics such as geodetic measurements of the Earth, searches for dark matter and investigations into possible long-term variations of fundamental physics constants.

Finding the most likely path to a set of failure states is important to the analysis of safety critical systems that operate over a sequence of time steps, such as aircraft collision avoidance systems and autonomous cars. In many applications such as autonomous driving, failures cannot be completely eliminated due to the complex stochastic environment in which the system operates. As a result, safety validation is not only concerned about whether a failure can occur, but also discovering which failures are most likely to occur. This article presents adaptive stress testing (AST), a framework for finding the most likely path to a failure event in simulation. We consider a general black box setting for partially observable and continuous-valued systems operating in an environment with stochastic disturbances. We formulate the problem as a Markov decision process and use reinforcement learning to optimize it. The approach is simulation-based and does not require internal knowledge of the system, making it suitable for black-box testing of large systems. We present different formulations depending on whether the state is fully observable or partially observable. In the latter case, we present a modified Monte Carlo tree search algorithm that only requires access to the pseudorandom number generator of the simulator to overcome partial observability. We also present an extension of the framework, called differential adaptive stress testing (DAST), that can find failures that occur in one system but not in another. This type of differential analysis is useful in applications such as regression testing, where we are concerned with finding areas of relative weakness compared to a baseline. We demonstrate the effectiveness of the approach on an aircraft collision avoidance application, where a prototype aircraft collision avoidance system is stress tested to find the most likely scenarios of near mid-air collision.

Diagnosis of COVID-19 at point of care is vital to the containment of the global pandemic. Point of care ultrasound (POCUS) provides rapid imagery of lungs to detect COVID-19 in patients in a repeatable and cost effective way. Previous work has used public datasets of POCUS videos to train an AI model for diagnosis that obtains high sensitivity. Due to the high stakes application we propose the use of robust and explainable techniques. We demonstrate experimentally that robust models have more stable predictions and offer improved interpretability. A framework of contrastive explanations based on adversarial perturbations is used to explain model predictions that aligns with human visual perception.

Estimations of human joint torques can provide quantitative, clinically valuable information to inform patient care, plan therapy, and assess the design of wearable robotic devices. Standard methods for estimating joint torques are limited to laboratory or clinical settings since they require expensive equipment to measure joint kinematics and ground reaction forces. Wearable sensor data combined with neural networks may offer a less expensive and obtrusive estimation method.We present a method of mapping joint torque estimates obtained from motion capture and ground reaction forces to wearable sensor data. We use several different neural networks to learn the torque mapping for the ankle joints during standing, walking, running, and sprinting. Our results show that neural networks that consider time (recurrent and long short-term memory networks) outperform feedforward network architectures, producing results in the range of 0.005-0.008 N m/kg mean squared error (MSE) when compared to the inverse dynamics model on which it was trained. As a point of reference, the typical measurement errors from inverse dynamics models are in the range of 0.0004-0.0064 N m/kg MSE. Errors tended to increase with locomotion speed, with the highest errors during sprinting and the lowest during standing or walking. Future work may investigate model generalizability across sensor placements, subjects, locomotion variants, and usage duration. The proposed method relies on learning from a motion capture dataset, but once the model is built, the method uses wearable sensors that enable torque estimation without the motion capture data. These methods also have potential uses for the design and testing of wearable robotic systems outside of a laboratory environment.

Compared to other major medical imaging modalities such as X-ray, computed tomography (CT), and magnetic resonance imaging, medical ultrasound (US) has unique attributes that make it the preferred modality for many clinical applications. In particular, US is nonionizing, portable, and provides real-time imaging, with adequate spatial and depth resolution to visualize tissue dynamics. The ability to measure Doppler information is also important, particularly for measuring blood flows. The small size of US transducers is a key attribute for intravascular applications. In addition, accessibility has been increased with the use of portable US, which continues to move toward a smaller footprint and lower cost. Nowadays, some US probes can even be directly connected to a phone or tablet. On the other hand, US also has unique challenges, particularly in that image quality is highly dependent on the operator’s skill in acquiring images based on the proper position, orientation, and probe pressure. Additional challenges that further require operator skill include the presence of noise, artifacts, limited field of view, difficulty in imaging structures behind bone and air, and device variability across manufacturers. Sonographers become highly proficient through extensive training and long experience, but high intra- and interobserver variability remains. This skill dependence has limited the wider use of US by healthcare providers who are not US imaging specialists. Recent advances in machine learning (ML) have been increasingly applied to medical US (Brattain, Telfer, Dhyani, Grajo, & Samir, 2018), with a goal of reducing intra- and interobserver variability as well as interpretation time. As progress toward these goals is made, US use by nonspecialists is expected to proliferate, including nurses at the bedside or medics in the field. The acceleration in ML applications for medical US can be seen from the increasing number of publications (Fig. 8.1) and Food and Drug Administration (FDA) approvals (Table 8.1) in the past few years. Fig. 8.1 shows that cardiovascular applications (spanning the heart, brain and vessels) have received the most attention, compared to other organs. Table 8.1 shows that pace of US FDA-cleared artificial intelligence (AI) products that combine AI and ultrasound is accelerating. Of note, many of the products have been approved over the last couple of years. Companies such as Butterfly Network (Guilford, CT) have also demonstrated AI-driven applications for portable ultrasound and more FDA clearances are expected to be published. The goals of this chapter are to highlight the recent progress, as well as the current challenges and future opportunities. Specifically, this chapter addresses topics such as the following: (1) what is the current state of machine learning for medical US application, both in research and commercially; (2) what applications are receiving the most attention and have performance improvements been quantified; (3) how do ML solutions fit in an overall workflow; and (4) what open-source datasets are available for the broader community to contribute to progress in this field. The focus is on cardiovascular applications (Section Cardiovascular/echocardiography), but common themes and differences for other applications for medical US are also summarized (Section Breast, liver, and thyroid ultrasound). A discussion is offered in Discussion and outlook section.

The capability to avoid other air traffic is a fundamental component of the layered conflict management system to ensure safe and efficient operations in the National Airspace System. The evaluation of systems designed to mitigate the risk of midair collisions of manned aircraft are based on large-scale modeling and simulation efforts and a quantitative volume defined as a near midair collision (NMAC). Since midair collisions are difficult to observe in simulation and are inherently rare events, basing evaluations on NMAC enables a more robust statistical analysis. However, an NMAC and its underlying assumptions for assessing close encounters with manned aircraft do not adequately consider the different characteristics of smaller UAS-only encounters. The primary contribution of this paper is to explore quantitative criteria to use when simulating two or more smaller UASs in sufficiently close proximity that a midair collision might reasonably occur and without any mitigations to reduce the likelihood of a midair collision. The criteria assumes a historically motivated upper bound for the collision likelihood and subsequently identify the smallest possible NMAC analogs. We also demonstrate the NMAC analogs can be used to support modeling and simulation activities.

In this paper we study dynamical distortion problems in future electrical energy systems with high renewable penetration. We introduce a new time-domain modeling of electrical energy systems comprising inverter-controlled distributed energy resources (DERs). This modeling is first used to quantify the relations between distortions and real/reactive power dynamics. Next, to ensure acceptable Quality of Service (QoS), a novel nonlinear distributed inverter control is introduced. Sufficient conditions are established for the guaranteed performance of the proposed control. These conditions further support the practical implementation of the derived controller. The effectiveness of this enhanced control is illustrated using simulations for the case of avoiding system instability during sudden grid reconfigurations. Simulations also show that distortions can be suppressed in systems with parallel-connected solar photovoltaics (PVs).