Publications
Multi-Agent Systems Collaborative Teaming (MASCOT) definition process to create specifications for Multi-Agent System (MAS) development
November 2, 2020
Conference Paper
Published in:
25th Intl. Command and Control Research and Technology Symp., ICCRTS 2020, 2-5 November 2020.
Topic:
R&D area:
R&D group:
Summary
The US Army envisions heterogeneous teams of advanced machines and humans that will collaborate together to achieve a common mission goal. It is essential for commanders to quickly and effectively respond to dynamic mission environments with agile re-tasking and computerized aids for plan definition/redefinition, and to perform some tasks with bounded autonomy. Workload constraints limit an individual's ability to concurrently control many platforms, so some mission segments many need to be autonomous or to be quickly selected via a tactics playbook. Denied environments also dictate the need for machine participants in some mission segments to be autonomous (or semi-autonomous). A Multi-Agent System (MAS) provides a natural paradigm for describing a system of agents that work together in such environments. An agent can be a human or machine, but is generally a machine. Creating MAS systems and requirements has proved to be a formidable task due to mission complexities, the necessity to deal with unforeseen circumstances, and the general difficulty of defining autonomous behaviors. We define a process called Multi-Agent Systems Collaborative Teaming (MASCOT) Definition Process that starts with a Subject Matter Experts (SME), produces a set of agent specifications, and derives system requirements in sufficient detail to define a MAS that can be modeled in a test-bed, used for facilitation of a safety analysis, and produced into an actual system. The MASCOT process also enables concurrent development of an effects based ontology. We demonstrate the MASCOT process on an example case study to show the efficacy of our process.
Summary
The US Army envisions heterogeneous teams of advanced machines and humans that will collaborate together to achieve a common mission goal. It is essential for commanders to quickly and effectively respond to dynamic mission environments with agile re-tasking and computerized aids for plan definition/redefinition, and to perform some tasks with...
READ MORE
Method to characterize potential UAS encounters using open source data
November 1, 2020
Journal Article
Published in:
Aerospace, Vol. 7, No. 11, November 2020, art. no. 158.
Summary
As unmanned aerial systems (UASs) increasingly integrate into the US national airspace system, there is an increasing need to characterize how commercial and recreational UASs may encounter each other. To inform the development and evaluation of safety critical technologies, we demonstrate a methodology to analytically calculate all potential relative geometries between different UAS operations performing inspection missions. This method is based on a previously demonstrated technique that leverages open source geospatial information to generate representative unmanned aircraft trajectories. Using open source data and parallel processing techniques,we performed trillions of calculations to estimate the relative horizontal distance between geospatial points across sixteen locations.
Summary
As unmanned aerial systems (UASs) increasingly integrate into the US national airspace system, there is an increasing need to characterize how commercial and recreational UASs may encounter each other. To inform the development and evaluation of safety critical technologies, we demonstrate a methodology to analytically calculate all potential relative geometries...
READ MORE
The 2019 NIST Audio-Visual Speaker Recognition Evaluation
November 1, 2020
Conference Paper
Published in:
The Speaker and Language Recognition Workshop: Odyssey 2020, 1-5 November 2020.
Topic:
R&D area:
Summary
In 2019, the U.S. National Institute of Standards and Technology (NIST) conducted the most recent in an ongoing series of speaker recognition evaluations (SRE). There were two components to SRE19: 1) a leaderboard style Challenge using unexposed conversational telephone speech (CTS) data from the Call My Net 2 (CMN2) corpus, and 2) an Audio-Visual (AV) evaluation using video material extracted from the unexposed portions of the Video Annotation for Speech Technologies (VAST) corpus. This paper presents an overview of the Audio-Visual SRE19 activity including the task, the performance metric, data, and the evaluation protocol, results and system performance analyses. The Audio-Visual SRE19 was organized in a similar manner to the audio from video (AfV) track in SRE18, except it offered only the open training condition. In addition, instead of extracting and releasing only the AfV data, unexposed multimedia data from the VAST corpus was used to support the Audio-Visual SRE19. It featured two core evaluation tracks, namely audio only and audio-visual, as well as an optional visual only track. A total of 26 organizations (forming 14 teams) from academia and industry participated in the Audio-Visual SRE19 and submitted 102 valid system outputs. Evaluation results indicate: 1) notable performance improvements for the audio only speaker recognition task on the challenging amateur online video domain due to the use of more complex neural network architectures (e.g., ResNet) along with soft margin losses, 2) state-of-the-art speaker and face recognition technologies provide comparable person recognition performance on the amateur online video domain, and 3) audio-visual fusion results in remarkable performance gains (greater than 85% relative) over the audio only or visual only systems.
Summary
In 2019, the U.S. National Institute of Standards and Technology (NIST) conducted the most recent in an ongoing series of speaker recognition evaluations (SRE). There were two components to SRE19: 1) a leaderboard style Challenge using unexposed conversational telephone speech (CTS) data from the Call My Net 2 (CMN2) corpus...
READ MORE
The 2019 NIST Speaker Recognition Evaluation CTS Challenge
November 1, 2020
Conference Paper
Published in:
The Speaker and Language Recognition Workshop: Odyssey 2020, 1-5 November 2020.
Topic:
R&D area:
Summary
In 2019, the U.S. National Institute of Standards and Technology (NIST) conducted a leaderboard style speaker recognition challenge using conversational telephone speech (CTS) data extracted from the unexposed portion of the Call My Net 2 (CMN2) corpus previously used in the 2018 Speaker Recognition Evaluation (SRE). The SRE19 CTS Challenge was organized in a similar manner to SRE18, except it offered only the open training condition. In addition, similar to the NIST i-vector challenge, the evaluation set consisted of two subsets: a progress subset, and a test subset. The progress subset comprised 30% of the trials and was used to monitor progress on the leaderboad, while the remaining 70% of the trials formed the test subset, which was used to generate the official final results determined at the end of the challenge. Which subset (i.e., progress or test) a trial belonged to was unknown to challenge participants, and each system submission had to contain outputs for all of trials. The CTS Challenge also served as a prerequisite for entrance to the main SRE19 whose primary task was audio-visual person recognition. A total of 67 organizations (forming 51 teams) from academia and industry participated in the CTS Challenge and submitted 1347 valid system outputs. This paper presents an overview of the evaluation and several analyses of system performance for all primary conditions in the CTS Challenge. Compared to the CTS track of the SRE18, the SRE19 CTS Challenge results indicate remarkable improvements in performance which are mainly attributed to 1) the availability of large amounts of in-domain development data from a large number of labeled speakers, 2) speaker representations (aka embeddings) extracted using extended and more complex end-to-end neural network frameworks, and 3) effective use of the provided large development set.
Summary
In 2019, the U.S. National Institute of Standards and Technology (NIST) conducted a leaderboard style speaker recognition challenge using conversational telephone speech (CTS) data extracted from the unexposed portion of the Call My Net 2 (CMN2) corpus previously used in the 2018 Speaker Recognition Evaluation (SRE). The SRE19 CTS Challenge...
READ MORE
Geospatial QPE accuracy dependence on weather radar network configurations
October 27, 2020
Journal Article
Published in:
J. Appl. Meteor. Climatol., Vol. 59, No. 1, 2020, pp. 1773-92.
Summary
The relatively low density of weather radar networks can lead to low-altitude coverage gaps. As existing networks are evaluated for gap-fillers and new networks are designed, the benefits of low-altitude coverage must be assessed quantitatively. This study takes a regression approach to modeling quantitative precipitation estimation (QPE) differences based on network density, antenna aperture, and polarimetric bias. Thousands of cases from the warm-season months of May–August 2015–2017 are processed using both the specific attenuation [R(A)] and reflectivity-differential reflectivity [R(Z,ZDR)] QPE methods and are compared against Automated Surface Observing System (ASOS) rain gauge data. QPE errors are quantified based on beam height, cross-radial resolution, added polarimetric bias, and observed rainfall rate. The collected data are used to construct a support vector machine regression model that is applied to the current WSR-88D network for holistic error quantification. An analysis of the effects of polarimetric bias on flash-flood rainfall rates is presented. Rainfall rates based on 2-year/1-hr return rates are used for a CONUS-wide analysis of QPE errors in extreme rainfall situations. These errors are then re-quantified using previously proposed network design scenarios with additional radars that provide enhanced estimate capabilities. Finally, a gap-filling scenario utilizing the QPE error model, flash-flood rainfall rates, population density, and potential additional WSR-88D sites is presented, exposing the highest-benefit coverage holes in augmenting the WSR-88D network (or a future network) relative to QPE performance.
Summary
The relatively low density of weather radar networks can lead to low-altitude coverage gaps. As existing networks are evaluated for gap-fillers and new networks are designed, the benefits of low-altitude coverage must be assessed quantitatively. This study takes a regression approach to modeling quantitative precipitation estimation (QPE) differences based on...
READ MORE
Automated posterior interval evaluation for inference in probabilistic programming
October 22, 2020
Conference Paper
Published in:
Intl. Conf. on Probabilistic Programming, PROBPROG, 22 October 2020.
R&D group:
Summary
In probabilistic inference, credible intervals constructed from posterior samples provide ranges of likely values for continuous parameters of interest. Intuitively, an inference procedure is optimal if it produces the most precise posterior intervals that cover the true parameter value with the expected frequency in repeated experiments. We present theories and methods for automating posterior interval evaluation of inference performance in probabilistic programming using two metrics: 1.) truth coverage, and 2.) ratio of the empirical over the ideal interval widths. Demonstrating with inference on popular regression and state-space models, we show how the metrics provide effective comparisons between different inference procedures, and capture the effects of collinearity and model misspecification. Overall, we claim such automated interval evaluation can accelerate the robust design and comparison of probabilistic inference programs by directly diagnosing how accurately and precisely they can estimate parameters of interest.
Summary
In probabilistic inference, credible intervals constructed from posterior samples provide ranges of likely values for continuous parameters of interest. Intuitively, an inference procedure is optimal if it produces the most precise posterior intervals that cover the true parameter value with the expected frequency in repeated experiments. We present theories and...
READ MORE
Failure prediction by confidence estimation of uncertainty-aware Dirichlet networks
October 19, 2020
Journal Article
Published in:
https://arxiv.org/abs/2010.09865
Topic:
R&D area:
R&D group:
Summary
Reliably assessing model confidence in deep learning and predicting errors likely to be made are key elements in providing safety for model deployment, in particular for applications with dire consequences. In this paper, it is first shown that uncertainty-aware deep Dirichlet neural networks provide an improved separation between the confidence of correct and incorrect predictions in the true class probability (TCP) metric. Second, as the true class is unknown at test time, a new criterion is proposed for learning the true class probability by matching prediction confidence scores while taking imbalance and TCP constraints into account for correct predictions and failures. Experimental results show our method improves upon the maximum class probability (MCP) baseline and predicted TCP for standard networks on several image classification tasks with various network architectures.
Summary
Reliably assessing model confidence in deep learning and predicting errors likely to be made are key elements in providing safety for model deployment, in particular for applications with dire consequences. In this paper, it is first shown that uncertainty-aware deep Dirichlet neural networks provide an improved separation between the confidence...
READ MORE
TCAS II and ACAS Xa traffic and resolution advisories during interval management paired approach operations
October 11, 2020
Conference Paper
Published in:
2020 AIAA/IEEE 39th Digital Avionics Systems Conf., DASC, 11-15 October 2020.
Topic:
R&D area:
R&D group:
Summary
Interval Management (IM) is an FAA Next-Gen Automatic Dependent Surveillance – Broadcast (ADS-B) In application designed to decrease the variability in spacing between aircraft, thereby increasing the efficiency of the National Airspace System (NAS). One application within IM is Paired Approach (PA). In a PA operation, the lead aircraft and trail aircraft are both established on final approach to dependent parallel runways with runway centerline spacing less than 2500 feet. The trail aircraft follows speed guidance from the IM Avionics to achieve and maintain a desired spacing behind the lead aircraft. PA operations are expected to require a new separation standard that allows the aircraft to be spaced more closely than current dependent parallel separation standards. The behavior of an airborne collision avoidance system, such as TCAS II or ACAS Xa, must be considered during a new operation such as PA, because the aircraft are so closely spaced. This analysis quantified TAs and RAs using TCAS II Change 7.1 and ACAS Xa software with simulated IM PA operations. The results show no RAs using either TCAS II Change 7.1 or ACAS Xa, negligible TAs using TCAS II Change 7.1, and acceptable numbers of TAs using ACAS Xa software during simulated PA operations.
Summary
Interval Management (IM) is an FAA Next-Gen Automatic Dependent Surveillance – Broadcast (ADS-B) In application designed to decrease the variability in spacing between aircraft, thereby increasing the efficiency of the National Airspace System (NAS). One application within IM is Paired Approach (PA). In a PA operation, the lead aircraft and...
READ MORE
Toward distributed control for reconfigurable robust microgrids
October 11, 2020
Conference Paper
Published in:
2020 IEEE Energy Conversion Congress and Exposition, ECCE, 11-15 October 2020.
Summary
Microgrids have been seen as a good solution to providing power to forward-deployed military forces. However, compatibility, robustness and stability of current solutions are often questionable. To overcome some of these problems, we first propose a theoretically-sound modeling method which defines common microgrid component interfaces using power and rate of change of power. Using this modeling approach, we propose a multi-layered distributed control: the higher control layer participates in dynamic power management that ensures acceptable voltage, while the lower layer stabilizes frequency by regulating the dynamics to the power determined by the higher layer. Numerical and hardware tests are conducted to evaluate the effectiveness of the proposed control.
Summary
Microgrids have been seen as a good solution to providing power to forward-deployed military forces. However, compatibility, robustness and stability of current solutions are often questionable. To overcome some of these problems, we first propose a theoretically-sound modeling method which defines common microgrid component interfaces using power and rate of...
READ MORE
Image processing pipeline for liver fibrosis classification using ultrasound shear wave elastography
October 1, 2020
Journal Article
Published in:
Ultrasound in Med. & Biol., Vol. 46, No. 10, October 2020, pp. 2667-2676.
Topic:
R&D area:
Summary
The purpose of this study was to develop an automated method for classifying liver fibrosis stage >=F2 based on ultrasound shear wave elastography (SWE) and to assess the system's performance in comparison with a reference manual approach. The reference approach consists of manually selecting a region of interest from each of eight or more SWE images, computing the mean tissue stiffness within each of the regions of interest and computing a resulting stiffness value as the median of the means. The 527-subject database consisted of 5526 SWE images and pathologist-scored biopsies, with data collected from a single system at a single site. The automated method integrates three modules that assess SWE image quality, select a region of interest from each SWE measurement and perform machine learning-based, multi-image SWE classification for fibrosis stage >=F2. Several classification methods were developed and tested using fivefold cross-validation with training, validation and test sets partitioned by subject. Performance metrics were area under receiver operating characteristic curve (AUROC), specificity at 95% sensitivity and number of SWE images required. The final automated method yielded an AUROC of 0.93 (95% confidence interval: 0.90-0.94) versus 0.69 (95% confidence interval: 0.65-0.72) for the reference method, 71% specificity with 95% sensitivity versus 5% and four images per decision versus eight or more. In conclusion, the automated method reported in this study significantly improved the accuracy for >=F2 classification of SWE measurements as well as reduced the number of measurements needed, which has the potential to reduce clinical workflow.
Summary
The purpose of this study was to develop an automated method for classifying liver fibrosis stage >=F2 based on ultrasound shear wave elastography (SWE) and to assess the system's performance in comparison with a reference manual approach. The reference approach consists of manually selecting a region of interest from each...
READ MORE