We discuss two specific short term aviation weather forecasts - convection and ceiling - to illustrate the issues that arise in thinking about the overall decision support system, key users, and training needed to generate benefits. We also consider reducing weather-related fatal accidents. Second, what is the preexisting "baseline" of aviation forecasts/decision processes that already exists to address the user needs? In most cases, there are already various weather information sources that can be viewed as providing a short term forecast (e.g., a Center Weather Service Unit (CWSU) meteorologist, persistence, or animation loops of the past weather). How well do we understand how the "baseline" forecast and the associated user decision support system operate? How will the new forecast and its decision support compare? What are the training implications if the new forecast is rather different than the "baseline"? Third, how will we measure the change in system performance? For example, if the new forecast claims to help reduce delays and/or accidents, how will one address differences in the weather between the "before" and "after" time periods? How will one determine whether the new forecast is in fact the key factor, if there was a change? The paper concludes with some suggestions for development and testing of new aviation forecasts to improve safety and reduce delays.