README.TXT for version 1.2 ------------------------------------- OVERVIEW TimeStat is a windows program designed for quick statistical study and preprocessing of time series data. Present features include: - spreadsheet like environment with rich subset of Excel 4.0 functionality, provided by Formula One VBX (TM) from Visual Tools, Inc. - fully functional edit bar for spreadsheet formulas. Let me emphasize that Formula One supports nearly all common Excel built-in functions. For example, I believe (but have not actually tested of course) that the Excel formulae presented in any issue of 'Technical Analysis of Stocks and Commodities' can be entered into TimeStat without modification (if not, please let me know). In effect, you are getting a FREE spreadsheet. - multiple document support with cut, paste, copy functions between documents and to/from other applications - transformations, calculations, and statistical analysis, including: FFT, correlation with time lag, power spectrum, sorting, and others - quick and easy charting of data; can open multiple charts for each spread sheet - reads Excel 4.0 worksheets or tab-delimited text files ***New in version 1.1: principal components analysis (PCA) and discrete wavelet transform (DWT) ***New in version 1.11: minor upgrade PCA output for full spreadsheet formula support; added 3 types of moving averages (simple, exponential, and adaptive) ***New in version 1.20: printing (both spreadsheet and graph, graph can also be sent to .WMF format file); sorting (row or column, multiple keys possible); fast Hartley transform; partial autocorrelation; BDS test for time series; fractional differencing; AND by popular request===> export whole or parts of spreadsheet to ASCII text file TimeStat is NOT: - a technical analysis program, even though it is possible to use it for that purpose by virtue of the built-in spreadsheet and edit bar - neural network or other types of forecasting programs, but it is meant to be useful for analyzing and preprocessing data bound for those programs - meant for commercial use; I use it as test bench to try out and to teach myself new techniques. But if someone can use it to make money, please contact me by private email. - going to cost you anything. It is freeware so I am not asking for any money. If you feel like rewarding me for my efforts, send me a note with your high praises, criticisms, and suggestions, and then may be go out and donate some money or time to any charity that helps the poor. That way, we will all feel better. *******STANDARD DISCLAIMERS*********** TimeStat is freeware, and I absolutely make no guarantee or warranty regarding its suitability for anything. Use at your own risk. I, my software suppliers, the authors of BDS test, or anyone else you can think of can never be held responsible for any possible damage or loss resulting from use of this software. ******************************************* INSTALLATION AND USAGE Decide where on your hardisk you would want this program to reside, create new directory if necessary. The program and all supporting files take about 1.5 MB. Copy all files to a single directory on your hardisk. That is it. If your want, the .VBX and .DLL and GSW.EXE files can be transferred to your WINDOWS\SYSTEM\ subdirectory as recommend by Microsoft for common installations. But that makes it complicated to uninstall (erase) the program and its supporting files. Run the program from Program Manager (choose File/Run then enter full path name), or File Manager (double click TIMESTAT.EXE), or you can drag the program from File Manager to an existing program group inside Program Manager, thereafter just double click on TIMESTAT.EXE's icon. I do not have the time to write documentation or a decent on-line help file. The online help is really spare. The TUTORIAL.TXT is what I can offer at this time. It takes you through some operations which should quickly convey the necessary ideas. The 'Help' menu provides some paltry guidance, but nowhere near enough. After writing the tutorial for several weeks, the now 27kB document has actually become somewhat educational. I hope people will find it useful and may be even contribute to it. The graph control uses GSW.EXE, a clumsy graphics server which sometimes may fail to close after you quit TIMESTAT.EXE. Just click on its icon and choose 'Close' on the pop menu, then answer 'yes' to closing dialog. Right now the program will only save in Excel 4.0 worksheet format. Sorry. I put an 'Export ASCII...' option under the 'File' menu but it is disabled because I have not had time to implement it. Will do so soon. For now, you can always use the good old cut and paste technique to copy and transfer data to an ASCII file editor. I plan to make this program OLE compatible which will also help. NOTES I would appreciate suggestions and bug reports. I wrote this program using Visual C++ 1.51. I use it as a test bench for new analysis/processing methods and techniques. I can do so simply by adding a new menu item and write a new member function for the document class. Therefore I am constantly adding new features and changes. The calculation speed seems to be Ok on most machines I tried, but I may be spoiled by my 486/66 with 24MB ram, running Windows NT 3.5 from a PCI hardisk; amount of RAM seems to be the most important factor. I was able to test this program successfully on a 386sx with 4MB ram, even though it was dog slow. Don't even bother with 286 or below. Version 1.11 and before does not require co- processor; but will use it if present. Starting with version 1.2, I have compiled the code using inline 80x87 coprocessor code. I believe this is not too restricting ever since I saw Pentium machines on sale for $1700 or less, besides, the BDS test really needs the coprocessor boost. The spreadsheet can handle up to 256 columns and 32,768 rows. But the graph.vbx that Microsoft supplied with Visual Basic 3.0 has an incomprehensible limit of 3800 points per graph. Which means if you want to plot 4 data sets (4 curves) on a graph, each dataset is limited to a maximum of 3800/4=950 points. Other features are also clumsy and limited. I have not implemented the UNDO editing feature, so take care when working with data. The program will not write over your data file unless explicitly instructed to do so by File/Save or File/Save As. A word about ERROR checking. I have made some reasonable effort to provide error-checking in the program, such as not feeding empty data sets or sets with 1 point to FFT. However, this program does assume that the user exercise some common sense. Using this program without some background and common sense can result in, at best, the old garbage in garbage out, or at worse, the dreaded general protection fault (GPF ) and all its lovely side effects. My NT system has survived many abuses during development, Windows 3.1 may not be so tolerant. I have not tried this program on Win'95, but there should not be any problems. At one point, the discrete wavelet transform (DWT) feature occasionally caused 'Out of stack space' error messages. I was not able to reproduce the error during subsequent testing, and I did allocate plenty for stack space. I have not tracked down the problem. In all cases of this error, I observed that re-running the program did not reproduce the error. The BDS test is very calculation intensive and somewhat fragile. I sacrificed speed for conservative memory usage; a faster version uses large amount of memory and I would only trust it in a 32-bit environment. The most common problem seems to be underflow. The program allow embedding dimensions of up to 10 but I would not try more than 8, and 5 works fine most of the time. FUTURE WORK (WISH LIST) All functions are currently linked statically to the main program. It probably makes more sense for them to reside in one or more separate DLLs. The advantage would be that the DLL's can then be used in Visual Basic and Excel. My list of to-do's include setting up the appropriate Excel VBA's to call such DLL's after they have been debugged and tested in TimeStat. I have only scratched the barest surface of the wavelet transform, an extremely rich, and interesting field. Future possibilities will include: user constructed bases, more threshold options, and wavelets on the interval. Whether I release new versions depend on interest and encouragement from others, on my available time. Suggestions are DEFINITELY welcome. If you have something in mind for me to code and add to this program, I'll be glad to consider giving it a shot as long as I understand it. Right now I myself have many ideas waiting for implementation. If I keep adding to TimeStat, this program may even become something useful someday. With wide availability of Windoze '95, this most likely marks the end of the line for TimeStat in 16-bit world. I am moving on to Visual C++ 4.0 and 32-bit programming (unless of course there is a huge public outcry and demonstrations outside my door denouncing this move...). I have the 32-bit spreadsheet and graphing OCX controls ready (full Excel 7.0 workbook compatibility, 3D scatter and surface plots with rendering, etc). OLE awareness is planned. I have the calculation code ready for many new features such as ARMA regression, genetic algorithm optimizer; as usual the user interface will require some thoughts and time. No firm schedule for future versions of course. ACKNOWLEDGMENT Robert Davies' versatile NEWMAT08 C++ matrix package simplifies much of my work. The package also provides the FFT routine used in TimeStat. He has given permission for me to use and distribute it as part of TIMESTAT (all bugs should still be reported to me). (Side note: I started off using Numerical Recipes routines, but I discovered that their routines are not public domain and require licenses for distribution. In my opinion, their FFT routines are not as good as Robert's.) This C++ matrix package is currently available by ftp at oak.oakland.edu and its mirrors, under directory msdos/cpluspls/newmat08.zip. The wavelet transform code is adapted from the Imager Wavelet Library written by Bob Lewis at University of British Columbia. The library and excellent lecture notes are available by ftp from ftp.cs.ubc.ca/pub/local/bobl/wvlt/ or on WWW at http://www.cs.ubc.ca/nest/imager/contributions/bobl/wvlt/top.html. BDS code kindly provided by Prof. Blake LeBaron of University of Wisconsin. The BDS quantile tables are reproduced, with permission from MIT Press, from the book "Nonlinear Dynamics, Chaos, and Instability" by Brock, Hsieh, and LeBaron 1993. The financial data is provided by NeuroVe$t Journal, P.O. Box 764, Haymarket, VA 22069-0764, voice (703)754-0696, fax (703)753-2634. Future versions of TimeStat may be announced in this journal. The adaptive moving average is featured in July 1995 issue of NeuroVe$t. Many thanks to editor, Randall Caldwell, for useful advices and constant encouragement. Questions, comments, bug reports welcome: Kenneth Lin kylin@onnes.physics.sunysb.edu 4 Hilton Court mmxp40a@prodigy.com Stony Brook, NY USA 11790-2354 (516)751-3101