Compliance Tests

1      Contents. 1

2      Table of Tables. 1

3      Compliance Testing Overview.. 3

3.1       SAPI Compliance Required Tests  3

3.2       SAPI Compliance Feature List Tests  3

3.3       Minimum Requirements  3

4      Using the compliance testing tool4

5      Compliance Tests. 5

5.1       Test Result Log:5

6      Compliance Testing Configuration Options. 5

6.1       SAPI 5.0 Compliance Testing Application Toolbar6

6.2       SAPI 5.0 Compliance Testing Application Menu Choices  7

6.3       SAPI 5.0 Compliance Testing Logging Options  9

6.4       SAPI 5.0 Compliance Testing Run Options  10

6.5       SAPI 5.0 Compliance Test Selection Options  11

7      SAPI Compliance: SR.. 13

7.1       Required Tests  13

7.2       Feature Tests  23

7.3       SR Sample Engine  27

7.4       Compliance Test Customization  29

7.5       Multilingual Support29

7.6       OS Language Incompatibility  31

8      SAPI Compliance: TTS. 32

8.1       Required Tests  32

8.2       Feature Tests  34

8.3       TTS Sample Engine  35

8.4       Multilingual Support36

8.5       OS Language Incompatibility  43

Table 1: Events Compliance Test14

Table 2: Lexicon Compliance Test16

Table 3: Command and Control Compliance Test20

Table 4: Required Compliance Tests. 22

Table 5: Events Feature Compliance Test23

Table 6: Grammar Feature Compliance Test25

Table 7: Feature Compliance Tests. 26

Table 8: Sample Engine Required Compliance Test results. 28

Table 9: Sample Engine Feature Compliance Test results. 28

Table 10: Strings to be localized. 31

Table 12: Speak Flag Tests. 32

Table 13: Speak Tests. 33

Table 14: Lexicon Tests. 33

Table 15: SAPI XML tests. 33

Table 16: Events Tests. 34

Table 17: Sample Engine Required Test Results. 36

Table 18: Sample Engine Feature List Test Results. 36

Table 19: Strings to be localized for compliance tests. 42

Table 21: Required Compliance Tests Failed. 43

Table 22: Feature Compliance Tests Not Supported. 43

This paper, directed toward engine vendors, describes the SAPI 5.0 compliance testing tool by answering the following questions:

·         What does SAPI compliance for SAPI 5.0 imply?

·         What are the SAPI compliance tests?

·         What does each test look for?  

The goals of the compliance tool are to help engine vendors test their speech engines for SAPI compliance and port these speech engines to SAPI 5.0. The tests also help vendors to support various SAPI features that are not required for compliance. These tests do not test the speech or performance quality of the engines.  All compliance tests assume that SAPI will do parameter validation, and as such, they do not check the engine’s ability to handle invalid parameters such as null, bad pointers, or values out of range.

To run the compliance tests, the SPcomp.exe tool is used and either the Text-to-Speech (TTS) or the Speech Recognition (SR) test suite is selected. This tool generates a log report indicating the results of the compliance tests.

There are two types of SAPI 5 compliance tests:

1)      required tests

2)      feature list tests

The compliance tests do not necessarily test the DDI directly, instead, the use the SAPI API function calls to test the engine’s response to the DDI. The default engine is always used as the engine in the compliance test. Currently, the supported languages for the compliance tests are English, Japanese and Simplified Chinese[1]. Please check Microsoft® Speech.NET Technologies for language pack updates and information. 

3.1       SAPI Compliance Required Tests

The results of the required tests are of a pass/fail nature. These tests were designed to help an engine reach a minimal amount of functionality with the SAPI DDI layer. In order to be SAPI compliant, the engine must pass all required SAPI tests.

3.2       SAPI Compliance Feature List Tests

The results of the feature list tests are either “Supported” or “Unsupported”. Feature list tests were designed to help engine vendors port advanced features to SAPI 5.0. To be SAPI compliant, the engine does not need to pass any feature list test, although it is recommended that all features be implemented if possible.

3.3       Minimum Requirements

The minimum requirements for speech recognition for dictation are a 200 mhz Pentium with 64 MB for win 95/98 or 96 MB for NT. The recommended computer is a 300 Mhz Pentium II with 128 megs of RAM or better.

The SAPI 5.0 compliance test tool, SPcomp.exe, enables you to load compliance test suites and determine the test result logging options. Please see Compliance Tests for more options.

The SPcomp.exe test tool creates the .pro file for a given test suite. Perform compliance tests by starting the SPcomp.exe application and loading a test suite from a .pro file. The .pro file loads the associated dynamic link library (.dll), which contains the SR or TTS compliance tests.

To start the SAPI 5.0 compliance test tool SPcomp.exe from Windows Explorer, double-click the compliance tool icon. Alternatively, you can perform each compliance test from the command line by running the compliance test tool and command line syntax.

For example, the command line syntax for running the SRcomp compliance test in the srcompreq.pro test suite is as follows:

C:> SPcomp.exe srcompreq.pro

SPcomp.exe srcompreq.pro

Starts the compliance test tool and loads the speech recognition (SR) required tests from the SRcomp.dll.

SPcomp.exe srcompopt.pro

Starts the compliance test tool and loads the speech recognition (SR) feature list tests from the SRcomp.dll.

SPcomp.exe ttscompreq.pro

Starts the compliance test tool and loads the text-to-speech (TTS) required tests from the TTScomp.dll.

SPcomp.exe ttscompopt.pro

Starts the compliance test tool and loads the text-to-speech (TTS) feature list tests from the TTScomp.dll.

The SAPI 5.0 compliance tests verify that you have successfully implemented the required features to be considered compatible with SAPI 5.0. Your engine must successfully complete each of the following four compliance tests with 100 percent pass rate to be compliant with SAPI 5.0.

1.      srcompreq.bat
  —Speech Recognition (SR) required test batch file.

2.      srcompopt.bat
  —Speech Recognition (SR) feature list test batch file.

2.      ttscompreq.bat
  —Text-to-speech (TTS) required test batch file.

4.      ttscompopt.bat
  —Text-to-speech (TTS) feature list test batch file.

5.1       Test Result Log:

The SAPI 5.0 compliance tool generates a result log and you can configure it to display the result log information or you can save it to a file. The test result log contains pass or fail state information for each segment of the test suite. If a compliance test fails, you can review the result log to determine the origin of the failure.

Example result log with a 100% pass rate for all tests.

Total:

=============================================================================

     PASS    FAIL    SUPPORTED    UNSUPPORTED    ABORTED    SKIPPED

-----------------------------------------------------------------------------

     1       0       0            0              0          0

=============================================================================

Status: PASS

The SAPI 5.0 compliance test application user interface (UI) enables you to configure the testing options. The following section provides additional compliance test configuration information.

·         SAPI 5.0 Compliance Testing Application Toolbar

·         SAPI 5.0 Compliance Testing Application Menu Choices

·         SAPI 5.0 Compliance Testing Logging Options

·         SAPI 5.0 Compliance Testing Run Options

·         SAPI 5.0 Compliance Test Selection Options

6.1       SAPI 5.0 Compliance Testing Application Toolbar

The main window of the SAPI 5.0 compliance testing application contains a toolbar from which you can access the configuration options. Additionally, the configuration options are also available from the menu bar located at the top of the compliance testing application window.

Pause on an icon to display tooltip text. Click an icon to view the information associated with the feature.

Load the test DLL

Loads a test dynamic-link library (DLL).

You can run compliance tests using one of the following methods:

1.      From the SAPI Engine Compliance Tool, click File, and then click Load Test DLL.

2.      Load the test DLL into SPcomp.exe from the command line.
For more information, see Using the compliance testing tool.

Note: loading a compliance test with either method results in automatically unloading any previously loaded compliance tests.

Load the test settings

Loads one of the pre-configured test suites.

You can run compliance tests using one of the following methods:

1.      From the SAPI Engine Compliance Tool, click File, and then click Load Settings.

2.      Load the test DLL into SPcomp.exe from the command line.
For more information, see Using the compliance testing tool.

Note: loading a compliance test with either method results in automatically unloading any previously loaded compliance tests.

Save settings

Saves the configuration settings for the compliance test application.

Copy

Selects and copies content from the display log.

Clear Window

Clears the display contents of the result log.

Find

Searches for a specific word or phrase within the result log.

Find Next

Searches for the next occurrence of a specific word or phrase within the result log.

Run Test

Begins the compliance test.

Stop Test

Stops the compliance test.

Set Run Options

Configures the compliance test options.

Select Tests

Chooses which compliance test contained in the test suite to.

Set Logging

Determines location of the compliance test log information.

6.2       SAPI 5.0 Compliance Testing Application Menu Choices

The SAPI 5.0 compliance testing application configuration choices are accessible through the menu system. The following items are covered in this section:

·         File menu

·         Edit menu

·         Test menu

·         Options menu

·         Help menu

6.2.1        File menu

Click File to set configuration options to load settings, save settings, or load the appropriate test DLL. Use the arrow keys to view various menu choices. Press ENTER to select a menu choice.

6.2.2        Edit menu

Click Edit to copy text from the result log and search for text within the result log. Use the arrow keys to view various menu choices. Press ENTER to select a menu choice.

6.2.3        Test menu

Click Test to run the test or select a test. Use the arrow keys to view various menu choices. Press ENTER to select a menu choice.

6.2.4        Options menu

Click Options to view the various configuration settings. Use the arrow keys to view various menu choices. Press ENTER to select a menu choice.

6.2.5        Help menu

Click Help and then click About to display the SAPI 5.0 Engine Compliance Tool Version dialog box. Use the arrow keys to view the various menu choices. Press ENTER to select a menu choice.

6.3       SAPI 5.0 Compliance Testing Logging Options

From the Options menu, choose Logging Settings to set SAPI 5.0 compliance test result log configuration options.

Window

Displays the test result information in the main window of the compliance testing application.

Log File

Saves the test result information as text in a log file.

The log file is located at the same directory as SPcomp.exe tool and the file name will be the following style:

[email protected]

The numbers "442" in the file name are generated by the SPcomp.exe tool and will be incremented by one each time you restart SPcomp.exe tool and run the test. A new log file is generated each time you start SPcomp.exe tool and run a compliance test.

Detailed

Specifies detailed result log information.

Summary

Specifies summary result log information.

6.4       SAPI 5.0 Compliance Testing Run Options

From the Options menu, click Run Options to configure SAPI 5.0 compliance testing run options.

Random

Randomizes the test order.

Close after execution

Closes the compliance testing application after the test sequence.

Stress

This option should not be selected for compliance tests.

Run count

Specifies the number of interactions the selected test should run.

Disable screen saver

Disable the screen saver.

Quiet

Runs the selected test in quiet mode.

Random Seed

The random seed value set here is used for the next time you run the compliance test.

Note: When troubleshooting a failed compliance test, you need to enter the same seed value information that was used for the failed compliance test before you repeat the compliance test procedure.

You can obtain the compliance test seed value from the "Random Seed" field information in the [email protected] file that was generated during the unsuccessful compliance test.

6.5       SAPI 5.0 Compliance Test Selection Options

From the Test menu, click Select Test to configure SAPI 5.0 compliance test choices.

Test Cases

Displays the current test suite.

Selected Test Cases

Displays the current selected tests.

Add Case(s)

Adds test items to the list of selected test cases.
Alternatively, to add test cases, right-click the test case in the test case display window and click Add Item.

Remove Case

Removes the selected test case from the current test. However, removing the selected test does not affect the need to successfully pass this test case to satisfy SAPI compliancy.

Alternatively, to remove test cases, right-click the test cases in the selected test case display window and click Remove Case.

Remove All

Removes all test cases.

SAPI compliant SR engines must be able to perform the following[2]:

§         Generate certain SR events

§         Interact with the SAPI lexicon

§         Handle Command and Control (C&C) grammars

§         Generate Phrase Elements

§         Support auto pause on recognition

§         Support rule synchronization

§         Support multiple instances of the engine

§         Support multiple application contexts

7.1       Required Tests

7.1.1        Events

Events will be checked for with .wav files. The test will feed the wav file to the engine and expect a specific event notification to occur. Please note that whether or not the engine can fire a specific event depends on the confidence threshold of the engine. Engine vendors could change the .wav quality to meet their requirement.

For English:

Table 1: Events Compliance Test

7.1.2        Lexicon

It is expected that changes in the user and application lexicon will be synchronized with the engine both when the engine starts up and after it has loaded a command and control grammar.

Table 2: Lexicon Compliance Test

7.1.3        Command and Control Grammar

Testing the engine for grammar compliance is perhaps the most complex set of tests. The engine must process a grammar correctly.  Each test will use a grammar specifically tailored for the particular feature.

Table 3: Command and Control Compliance Test

7.1.4        Phrase Elements, Auto Pause, Rule invalidation, multiple instances and contexts.

Table 4: Required Compliance Tests

7.2       Feature Tests

Some of the features exposed through SAPI are useful from a competitive advantage point of view. Features are not required by SAPI compliance, but may be an attractive function for engine vendors to implement. SAPI features are:

§         Interference and hypothesis events

§         Dictation functionalities

§         Advanced command and control features

§         Command and control alternate

§         Engine properties

§         Inversed text normalization

7.2.1        Events

Events will be checked for with .wav. The test will feed the .wav  to the engine and expect a specific event notification to occur. Please note that whether or not the engine can fire a specific event depends on the confidence threshold of the engine. Engine vendors may change the .wav files if it is felt that the .wav quality does not meet their requirements (Refer to Section 7.4).

Table 5: Events Feature Compliance Test

7.2.2        Dictation functionalities

This the required features if Engine wants to support dictation grammar. This include some basic functionalities for dictation grammar. This includes lexicon, dictation tag, dictation alternates.

Table 5: Dictation Compliance Test

7.2.3        Grammar

Each test will use a grammar specifically tailored for the particular feature. Some tests would use dynamic grammar instead of the static grammar.

Table 6: Grammar Feature Compliance Test

7.2.4        Alternates, engine properties, inversed text normalization

Table 7: Feature Compliance Tests

7.3       SR Sample Engine

The sample engine is not fully SAPI compliant due to the fact that it does not have the full range of functionality that a true SR engine would have. Table 8 indicates which compliance tests will pass. Table 9 indicates which features are supported.

Table 8: Sample Engine Required Compliance Test results

Table 9: Sample Engine Feature Compliance Test results

7.4       Compliance Test Customization

Many of the tests do require that a specific recognition result be returned to verify proper handing of such things as the grammar format. To accommodate different engines variability with recognition of different voices and to support non-English engines, these tests will enable the engine vendor to supply a sound file that passes the test (Refer to Section 7.5). Since some tests might share the same .wav file, it is recommended to supply a .wav file with different name. Additionally the grammars can be changed to accommodate words that the engine is able to recognize better (Refer to Section 7.6). 

7.5       Multilingual Support

The compliance tests will tests engines for the supported languages[3]. To test an SR engine that uses another language, one must:

§         Ensure that the correct language pack is installed. For Windows 2000 and Millennium Edition, this may be done by installing the language pack from the Windows 2000 or Windows Millennium CD. For Windows 98 and Windows NT 4.0, install the language pack from the Windows Update web site.

§         Select the engine as the default engine using Speech Recognition tab in Speech properties.

§         Create and insert a string table in the sapi5sdk\tools\comp\sr\srcomp.rc that is localized for the language. (Refer to Table 10)

§         Create the .wav files[4] according to the new string table and place this  under the specified directory (according to the search path precedence (Refer to Section 7.5.2)).

§         Create and compile the appropriate XML files using a grammar editor and complier.

§         Include the CFG binaries into the .dll by importing the CFG file names into srcomp.rc[5].

§         Recompile the sr.dsp.

§         Run the compliance tests.

7.5.1        Example:

If  you want to add resource for test SoundStart for language 888:

1.      create and insert copy a string table in srcomp.rc for language 888.

2.      Change the string “IDS_WAV_SOUNDSTART” to the new .wav file you want to use.

3.      Insert the xml grammar file you want to use into the project. Modify the IDR_L_GRAMMAR reference to your cfg binary.

NOTE: If the default engine supports multiple languages, then the compliance test will only run on the first language specified in string “Language” under key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Speech\Recognizers\Tokens\MSASREnglish\Attributes. In other words you need to change the order of the languages in the attributes key under your speech recognizer token for each language you wish to test. (Refer to Section 7.6)

The strings that need to be localized are shown in Table 10.

Table 10: Strings to be localized

7.5.2        Search Path Precedence

The compliance tests use a search path precedence to find the various .wav files needed for the compliance tests. The order of search is:

§         current directory

§         ‘..\resources’,

§         ‘..\..\..\resources’

§         ‘..\..\resources’

If the compliance tests cannot find the .wav files in these directories, the test will pop up a dialog asking the user to enter the customized dir to open the file. The new directory will be added to the search order and the new search order will persist for the life of SpComp.

It is important to note that for languages that do not have a SAPI standard phoneme set (i.e. languages which are not supported in this version of SAPI), the engine will fail the following required compliance tests:

7.6       OS Language Incompatibility

The compliance tests are not based on the language of the OS. They are based on the first language in the token of the default engine. In other words, a Japanese engine on an English OS will cause the compliance tests to load the Japanese resources and run the compliance tests expecting a Japanese engine. In order to run the compliance test for an engine that has a different language than the OS, you will need to set the default engine on the Speech Recognition tab in Speech properties.

NOTE: If the default engine supports multiple languages, then the compliance test will only run the first language. In other words you need to change the order of the languages in the attributes key under your speech recognizer token for each language you wish to test. For example, if your engine token supports both Japanese and English, to test English, , the string “Language” must be like “409; 411” under HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Speech\Recognizers\Tokens\MSASREnglish\Attributes

To test Japanese, the string “Language” must be like “411; 409” under the same key.

SAPI compliant TTS engines must be able to perform the following:

§         Speak with SAPI defined speak flags

§         Return the supported audio output formats

§         Interact with the SAPI lexicon

§         Interpret SAPI XML tags

§         React to programmatic volume changes

§         React to programmatic rate changes

§         Synthesize certain SAPI events

§         Skip forward and backward through a segment of text

§         Support multiple instances

8.1       Required Tests

8.1.1        Speak

The speak tests test the engine ability to interact with ISpTTSEngine::Speak as well as process and react to actions. Before SAPI passes a speak call to the engine, it Query Interfaces (QIs) the engine object for ISpTTSEngine and ISpObjectWithToken interfaces. If either of these are not implemented correctly, the speak call will fail. If both of the QIs pass, then SAPI will pass the speak call to the engine. The speak call contains a number of speak flags. Table 1 describes the individual tests.

Table 12: Speak Flag Tests

The compliance tests also test a number of combination scenarios to help ensure that the engine is able to stop speaking and start speaking in various combinations that are shown in table 2.

Table 13: Speak Tests

8.1.2        Output Format

The output format test checks to ensure that the engine is capable of passing its supported output format to SAPI. This is testing the ISpTTSEngine::GetOutputFormat function.

8.1.3        Lexicon

The lexicon tests check the interaction of the engine and SAPI. These tests add a word to the user and application lexicons and check to ensure that the engine is able to detect the words in the lexicon and is using the word pronunciation from the user lexicon first, and then the application lexicon second if the word is present in both lexicons. There are two separate tests as shown:

Table 14: Lexicon Tests

8.1.4        XML Tags

The SAPI XML tags are required for compliance. The tags tests shown are required:

Table 15: SAPI XML tests

8.1.5        SetVolume

This test checks to see if the engine is capable of processing the volume change action. When the engine received this action, it should call the GetVolume function from SAPI to get the new volume, and reflect the change in the audio output.

8.1.6        SetRate

This test checks to see if the engine is capable of processing the rate change action. When the engine received this action, it should call the GetRate function from SAPI to get the new volume, and reflect the change in the audio output.

8.1.7    Events

The events test checks to ensure that the engine is writing the correct data, especially wParam and lParam, to the event structure. For the sentence boundary event, wParam is the character length of the sentence including punctuation in the current input stream being synthesized. lParam is the character position within the current text input stream of the sentence being synthesized. For the word boundary event, wParam is the character length of the word in the current input stream being synthesized. lParam is the character position within the current text input stream of the word being synthesized. Any leading and ending spaces will not be included in the length of the word or the sentence. There are three events which are required for SAPI compliance:

Table 16: Events Tests

8.1.8        Skip

This test examines the engine’s ability to interact with the skip action. Once the engine receives this action, it should call the ISpTTSEngineSite::GetSkipInfo function. After it has completed the skip, it should call the ISpTTSEngineSite::CompleteSkip function.

8.1.9    Multi-Instance

The multiple instances test checks to ensure that the engine can handle multiple calls at the same time from SAPI. The tests contains a total of 4 threads and each thread has its own ISpVoice object and the test runs a random combination of the following tests 20 times consecutively:

§         Speak

§         Skip

§         GetOutPutFormat

§         SetRate

§         SetVolume

§         Check SAPI required Event

§         XML Bookmark

§         XML Silence

§         XML Spell

§         XML Pron

§         XML Rate

§         XML Volume

§         XML Pitch

§         Real time Rate changes

§         Real time Volume changes

§         Speak Stop

§         Lexicon

§         XML context

§         Engine proprietary SAPI tags and other combination of XML tags

8.2       Feature Tests

Some of the features exposed through SAPI are useful from a competitive advantage point of view. Features are not required by SAPI compliance, but may be an attractive function for engine vendors to implement. SAPI features are:

§         Generation of phoneme events (determines if the engine can generate a phoneme event for a given string of text. The phonemes must correspond to the SAPI defined phonemes).

§         Generation of viseme events (determines if the engine can generate a viseme for a given string of text. The viseme must correspond with the SAPI defined visemes).

§         XML emphasis tag (the engine should change the volume, rate, or pitch of the audio rendered).

§         XML PartOfSp tag (the engine should handle the SAPI defined part of speech – the engine will need to implement this for the lexicon compatibility tests)

8.3       TTS Sample Engine

The sample engine is not fully SAPI compliant due to the fact that it does not have the full range of functionality that a true TTS engine would have. Table 6 indicates which compliance tests will pass. Table 7 indicates which features are supported.

Table 17: Sample Engine Required Test Results

Table 18: Sample Engine Feature List Test Results

8.4       Multilingual Support

To test an engine that uses language aside for the supported languages, one must:

§         Ensure that the correct language pack is installed. For Windows 2000 and Millennium Edition, this may be done by installing the language pack from the Windows 2000 or Windows Millennium CD. For Windows 98 and Windows NT 4.0, install the language pack from the Windows Update web site.

§         Select the engine as the default engine using the TTS tab in Speech properties.

§         Create a string table in the \sapi5sdk\tools\comp\tts\ttscomp.rc which is localized for the language. (Refer to Table 19) 

o       Go to ResourceView in the ttscomp workspace, right click the mouse on “String Table”, and select “Insert Copy”. The following window will appear. From the window, select the language that the engine supports, and then click OK.

o       Open \sapi5sdk\tools\comp\tts\ttscomp.rc to your editor and edit your language resources (Refer to Table 19), and then save ttscomp.rc

o       The following is an example how to support GetOutputFormat test in Korean using Microsoft FrontPage editor:

§         Open ttscomp.dsw and create a string table in Korean, save ttscomp.rc

§         Go to below table 19 and find strings used in GetOutputFormat test.  Only one string, IDS_STRING65, is found corresponding to the test.

§         Open ttscomp.rc in Notepad and find IDS_STRING65 under Korean resources

§         Launch Microsoft FrontPage and select File | New and Normal tab

§         Translate “This is the TTS Compliance Test” to Korean.

§         Select Preview tab, right click your mouse, select Encoding | Western European (Windows), and then cut/paste the string from MS FrontPage to IDS_STRING65 under Korean resources in your notepad

§         Save ttscomp.rc

§         Recompile the tts.dsp.

§         Run the compliance tests.

The strings that need to be localized are:

Table 19: Strings to be localized for compliance tests

It is important to note that for languages which do not have a SAPI standard phoneme set (i.e. languages which are not supported in this SAPI release), the engine will fail the following required compliance tests:

Table 21: Required Compliance Tests Failed

The engine will also fail the following feature tests:

Table 22: Feature Compliance Tests Not Supported

8.5       OS Language Incompatibility

The compliance tests are not based on the language of the OS. They are based on the first language in the token of the default engine. In other words, a Japanese engine on an English OS will cause the compliance tests to load the Japanese resources and run the compliance tests expecting a Japanese engine. In order to run the compliance test for an engine that has a different language than the OS, you will need to set the default engine on the Speech Recognition tab of Speech properties.