MES Test Manager® (MTest) Release Notes
Model Engineering Solutions GmbH | Waldenserstraße 2-4 | 10551 Berlin | Germany
Contact: +49 (0)30-2091 6463 0 | mtest@model-engineers.com
Authors: Tobias Schmidt | Martin Hill | Hartmut Pohlheim
For more information visit the MES Test Manager® Support Site
© 2008-2024: Model Engineering Solutions GmbH
Contents
- MTest v 7.8.4 (April 2024)
- MTest v 7.8.3 (January 2023)
- MTest v 7.8.2 (October 2022)
- MTest v 7.8.1 (July 2022)
- MTest v 7.8.0 (April 2022)
- MTest v 7.7.0 (December 2021)
- MTest v 7.6.0 (September 2021)
- MTest v 7.5.0.Linux (August 2021)
- MTest v 7.5.0 (June 2021)
- MTest v 7.4.0 (April 2021)
- MTest v 7.3.2 (March 2021)
- MTest v 7.3.1.Linux (February 2021)
- MTest v 7.3.0 (December 2020)
- MTest v 7.2.0 (September 2020)
- MTest v 7.1.0 (June 2020)
- MTest v 7.0.1 (February 2020)
- MTest v 7.0.0 (December 2019)
- MTest v 6.4 (September 2019)
- MTest v 6.3 (July 2019)
- MTest v 6.21 (May 2019)
- MTest v 6.2 (March 2019)
- MTest v 6.1 (December 2018)
- MTest v 6.0 (September 2018)
- MTest v 5.2 (June 2018)
- MTest v 5.1 (March 2018)
- MTest v 5.0 (December 2017)
- MTest v 4.4 (September 2017)
- MTest v 4.3 (June 2017)
- MTest v 4.2 (March 2017)
- MTest v 4.1 (December 2016)
- MTest v 4.0 (October 2016)
- MTest v 3.52 (August 2016)
- MTest v 3.5 (June 2016)
- MTest v 3.4 (March 2016)
- MTest v 3.3 (Dezember 2015)
- MTest v 3.2 (September 2015)
- MTest v 3.1 (June 2015)
- MTest v 3.0 (May 2015)
MTest v 7.8.4 (April 2024)
- Bug Fixes and Maintenance
Compatibility with Matlab 2023b
Small bugfixes to ensure compatibility with Matlab 2023b Generation of Test Project Protocol in Batch mode
The Test Project Protocol is now generated as intended as part of the Test Catalog when it is created
in Batch mode. White noise for input signals
New function MTmp_setNoiseInSignals has been added to manually add white noise to input signals, which
was previously only available on request.MTest v 7.8.3 (January 2023)
- Bug Fixes and Maintenance
This release mainly focuses on fixing bugs and improving robustness and user experience. These affect
various functionalities in MTest, e.g. the support of Matlab 2022b and an updated licence management.MTest now officially supports Matlab 2022b.
Note: Changes in TargetLink 2022-B mean that TargetLink logging must always be enabled when logging
signals in TargetLink models, regardless of which mechanism you want to use for this.
Otherwise, signals logged with Simulink will not be logged either. #10600 We fixed an issue where referenced models could not be selected as a test object. This occurred
mainly when there were only referenced models as subsystems at the highest level in the model. #10595 We updated the licence mechanism to reduce problems when other MES tools are running at the same time
in the same Matlab session.MTest v 7.8.2 (October 2022)
- Bug Fixes and Maintenance
This release mainly focuses on fixing bugs and improving robustness and user experience. These affect a
range of functionalities in MTest, e.g. licence management, coverage and testbed generation. #10129 The Lismo-based license management in MTest is now the same for Linux and Windows operating systems.
The MTest Specification Editor is currently limited to use on Windows operating systems. #8531, We fixed an issue where test sequence coverage results in the aggregated coverage were not correctly
#10013 removed, even though the sequences were properly removed via the regarding menu entry. This
particularly affected the deletion of test groups.
In addition, Simulink coverage data is now also be deleted correctly when building a new testbed and
importing a test sequence, regardless of the currently active simulation mode. #8172 We fixed an issue where the bus' signals were duplicated in the interface if they were selected
by more than one bus selector. #8062 We fixed an issue with the automatic detection of multiple bus object instances at multiple outports.
In this case the bus renaming is enabled by MTest. We fixed an issue caused by command behavior changes in the latest Matlab version.
As a result, MTest now also supports Matlab 2022a.MTest v 7.8.1 (July 2022)
- Bug Fixes and Robustness Improvements
This release mainly focuses on fixing bugs and improving robustness and user experience. These affect a
range of functionalities in MTest, e.g. test bed generation, project configuration and test execution. #9683: We added additional modes for the BusElementRenaming option. This option adds the bus structure
to bus signal names as a prefix, in case different buses contain signals with identical names.
If the bus structure has too many hierarchies, the two new options prevent the signal names from
exceeding the maximum number of 73 characters, by shortening the prefix at the beginning or the end
A detailed description can be found in the configuration documentation. #9804: We fixed an issue related to the InheritInputDataTypes option. If the input interface of a test object
contains vector signals in buses, the signals are no longer converted to a specific data type, in case
this option is activated. The "Data Type Override (True Doubles)" option is now reset in the test bed after each batch run.
Rebuilding the test bed is no longer necessary, when the option is deactivated again. #9911: We have accelerated the textual output of vectorial values, which means that reports and catalogs are
now generated faster. This will be particularly noticeable when outputting larger vector parameters. #10013: If a test sequence is deleted using the menu, the associated coverage data is now also properly deleted.
This affects both the coverage data of the test sequence itself, and the aggregated coverage data for
the test object, ensuring that no inconsistent coverage results are shown in the reports.
In order to get the correct aggregated coverage values of the test object, only one test sequence has to
be executed again, with coverage enabled. #10035: We fixed an issue where the user was not correctly informed if an error occurred during testbed
preprocessing. The batch execution now displays a NOT OK as the result. In addition, there is no longer an attempt to carry out further actions for the test object in batch
mode, if the test bed could not be created correctly. If an error occurs during the execution of a test case when determining the model coverage, the user
is now immediately informed of this error. Previously, the test case ran without coverage instead,
without reporting that coverage data was missing.We have updated various chapters of the user guide for better understanding.
MTest v 7.8.0 (April 2022)
- Improved Support for AUTOSAR Models
The use of Embedded Coder AUTOSAR models is better supported now. If the model is recognized as an
AUTOSAR model, the model settings are now also transferred to the test bed. This feature is only available
for MATLAB 2018b and newer and not in combination with CTC Coverage. (#9785) - Test Case Generation for More Complex Requirements
We fixed an issue resulting in test case generation sometimes failing to generate test sequences when the
condition of the requirement consisted of multiple AND and OR conditions linked together by parentheses.
Test sequences are now generated for this type of requirement. (#9488) - TargetLink Models with Simulink Test Objects
The test bed generation of test objects that are in a TargetLink model but consist exclusively of Simulink
elements with virtual buses is fully supported now. (#9787) - Bug Fixes and Stability Optimization
This release also focused on fixing bugs, optimizing stability, and improving user experience. These affect a
wide range of functionalities in MTest, from generating test cases and the test bed to simulation and result
evaluation. #9857: Very long signals are no longer an issue for the assessment analyzer. Previously, sequences with very
long signals could cause MTest and MATLAB to freeze when plotting graphs of passed and failed
regions in the sequence-assessment view. This has been fixed now. #9854: We fixed a runtime error that could occur when generating the interfaces file of MARS and MTCD and was
caused by signals having the same name as protected variables, for example PI. Please note: Some names
of MATLAB function, MTCD functions, or special value names in MATLAB are still not valid for use as
signal names, for example PI, nan, inf. #9853: We fixed a runtime error that occurred when the model's path changed or was temporarily unavailable. This
prevented further processing of MTest, even if the model was not needed for this. The adaptation was made
for Windows and Linux systems. #9852: We fixed an issue where an empty logged signal resulted in signal logging not working at all. This can
happen when signals are logged in a subsystem that is not triggered in the test sequence. #9851: We fixed an issue with the CTC configuration where an additional TargetLink default configuration file
caused the coverage not to be performed correctly. #9809: We fixed a runtime error that could occur when the Lismo settings file could not be read. A useful error
message has now been added to the reported error, which helps users to find the cause of the problem
more quickly. #9796: We fixed a bug for simulations with variable step size where the simulation and the result data output
was configured by the user for different workspaces. This is particularly interesting for TargetLink
models on newer MATLAB versions in which the simulation workspace is always forced to the base workspace. #9777: We fixed an error that prevented the automated import of the MTCD test sequences within the batch run.
This error occurred in particular for TargetLink projects and when using MTCD in Excel. #9740: We fixed a bug where input signals might be assigned the wrong data type if a signal name appears twice
in the inports. Please note: If several inputs of a test object have the same name, this is only
permissible if they are exactly the same signal! This problem usually only occurs with bus signals.We fixed an issue resulting in unnecessary warnings when importing MTCD test sequences from an Excel file.
We removed unnecessary warnings related to ignoring subsystems in coverage. These warnings are only of
interest when running the simulation, but do not provide any useful information when collecting the result data
or generating the reports.MTest v 7.7.0 (December 2021)
- Support for Enumeration Definitions in Simulink Data Dictionary
- Enumeration Definitions with Numerical Values in MTCD & MARS Interface File
In addition to the symbolic name, the numerical values of the enumerations
are now also written in the MARS and MTCD interface file with the extension *.io.
Likewise, numerical values can be assigned for manually added enumerations too.
This makes it possible to use the symbolic name of enumerations instead of the
numerical value in MTCD and in MARS for signals and parameters, if a model has
SLDDs set as workspace. It may be necessary to carry out a new, once-off interface
analysis and a re-import of the test sequences.- Proper Closing of Simulink Data Dictionaries
When creating a testbed, MTest also creates a copy of the original SLDD from the
model. This ensures that the test does not alter the original model and that a
reset of the test data can be done between simulations. If enumerations are used
within the SLDD, the alternating access to the SLDD of the model and the
test-bed often leads to problems in MATLAB, since enumeration definitions may
only exist once.
To minimize these conflicts, a new option has been introduced to close the model
and the testbed with their associated SLDDs whenever necessary. This option is
available for Embedded Coder models in MATLAB 2018b and newer, as the correct
functioning can only be guaranteed here. The new parameter for this is "SupportEnumDefinitionsInSLDD" and can be found in
the menu under "Manage Project Configuration" or in "MTest_Configuration.m".- Undefined Areas in Logged Signals (#9343 & #9389)
During signal logging there may be time intervals, where signals are not defined
e.g. due to logging of states in state machines or triggered subsystems. If no
values are available for time steps, these are provided in MTest by default with
a NaN value so that all the signals have the same format for further processing.
In this way, the results show only the values that the simulation actually determined.
However, writing assessments or interpretation of the data can be easier if NaN values
are replaced by the omitted numerical values in the signals. Therefore, these
undefined areas can now be filled in by holding the last valid value to improve
evaluation and reporting. The new parameter for this is "LoggedSignalsFillNaNs" and can be found in
the menu under "Manage Project Configuration" or in "MTest_Configuration.m". - Bug Fixes
#9532: Improved error message that shows during report generation caused by using
project paths that are too long. #9517: Support for the TargetLink properties 'Inherit' and 'Uniform Elements'
on bus signals when building a test-bed.MTest v 7.6.0 (September 2021)
- Handling of MARS/MTCD Interface-Files
The *.io files are now being updated together with the interface analysis
without needing to open the Editor. This ensures the correct interface
information in particular during the batch runs. - Manual Enumeration Definitions
Now enumerations are not deleted from the *.io files if they were not
found during interface analysis. This allows the manual definition of enumerations
next to the automatically inserted definitions.
Note: Outdated enumeration definitions should be deleted manually when necessary. - TargetLink Bus Vector Handling
The interface analysis is now able to handle TargetLink bus vectors. (#9167) - Coverage Filter Information in Reports
The test report now shows if a coverage filter is configured. (#7767) - Protected Referenced Models
During testbed generation, protected referenced models are now included
correctly into the testbed. (#9229) - Logged Signals in Referenced Models
This fix resolves a problem regarding referenced models, when signal data is
logged in state machines. - Further Bug Fixes
#7361: In case the user aborts the batch execution manually they
will now be informed that the aggregated model coverage
data is not available.
#7356: The execution date in the main GUI now corresponds to the
timestamp of the output data for the currently selected
execution mode. It is marked as outdated if the input data
is newer than the simulation date.MTest v 7.5.0.Linux (August 2021)
- Beta Version for Use of MTest in Linux CI Environments
In the context of CI applications, MTest can now also be used on
Linux operating systems, especially for the execution of CI batch
runs. The following functional limitations should be noted: - Testing of TargetLink models is not supported.
- Project paths must not contain backslashes.
- The older Excel format *.xls is not supported. Prior to
import, please convert your requirement and/or MTCD test
specification documents, originally saved as *.xls files,
into *.xlsx files.
- Collection of code coverage data with Testwell CTC++ is
not supported. However, code coverage with Mathworks'
Simulink Coverage Toolbox is working.
- The application of custom data logging configurations
(originating from modified testbeds) cannot be performed.
- Excel documents cannot be opened from within MTest.
- The MTest Specification Editor is not part of the prototype
release and therefore cannot be used.
- Please note that it might be possible that currently limited
functionality, e.g. support of *.xls files, may lead to
MATLAB errors due to missing interceptions in all misuse
cases.
- The default build directory ('C:\work\MTest_Batch_Projects\')
was not adapted in MTest_Demo_Build_Autopilot.m (Line 114).We encourage all Linux users to report any further issues.
MTest v 7.5.0 (June 2021)
- Provision of Enumerations for Use in the MTest Specification Editor
With this version, MTest is able to automatically provide the test
object's interface signals and workspace parameters of type
enumeration (including their members) for use in the
MTest Specification Editor. Both signals and parameters are now
automatically written to the *_interface.io file.
Note: Enumerations defined in Simulink Data Dictionary cannot be
used in MTCD and MARS yet. Please keep using their respective
numeric values instead. - Possibility to Provide a Default Configuration
There is now a way to provide a standard configuration for the
MTest projects that need to rely on one. For this purpose, the
function <MTESTROOT\bin\AddOn\ConfigMan\MTest_getConfigData.m> is
delivered as source code with this release. The values of the
variable "DefaultValue" can be adapted individually for each
configuration parameter in the source code.
Please note: Changes are only allowed to the variable
"DefaultValue". Any changes to other variables (e.g. Section,
Name, Levels) may cause MTest to malfunction. - New Option for Signal Data Type Analysis of TargetLink Subsystems
In this version of MTest a new configuration option has been added
to specify at which interface level the signal data types of
TargetLink subsystems are to be analyzed: either at the outer
boundaries of the TargetLink intermediate layers or directly at
the TargetLink ports of the actual TargetLink subsystem.
For this purpose we introduced the parameter
TargetLinkPortAnalysisSource. It can have the following values: 0: The interface data is analyzed at the Simulink ports of the
TargetLink subsystem, i.e. at the ports located within the
Simulink subsystem which is two levels above the actual
TargetLink subsystem.
1: The interface data is analyzed directly at the TargetLink
ports of the TargetLink subsystem.
The default value is 1. - Support for Displaying Complex Signals in the Assessment Analyzer
In order to display complex (mathematical) signals
(e.g. a = sqrt(-1)) in the Assessment Analyzer, they are now split
into their real and imaginary parts and displayed separately. - Further Bug Fixes
#7929: This fix corrects a misbehavior where after deleting a MARS
requirement, the associated generated assessment persisted.
Now it is ensured that only generated assessments based on
valid MARS requirements exist.
#8557: This fix ensures that the port names are used for output
buses if the setting of the configuration parameter is
"SourceForSignalNaming = 1". In the past it happened that
despite this setting the signal names were applied for use
in MTest.
#8671: This fix ensures that generated code is properly simulated
during Embedded Coder SiL. Previously, it could happen
that in SiL mode the MiL test bed was mistakenly used for
simulation. Now, the batch test ensures that, if no code is
currently available, it is also generated before a SiL
simulation, even if the "Code Generation" batch action is
not active.
#8992: This change fixes the bug where the batch test crashes if
there is no interface file and the batch action
"Regenerate Testbed" is selected.
#9014: This fix suppresses the output of a specific warning
message when applying model coverage filtering to items
inside linked libraries. In addition, improved error
handling has been introduced in case the above filtering is
applied in conjunction with activated TargetLink code
coverage filtering.
#9063: This change ensures that the user gets an explicit feedback
in case the conversion of the system under test to a
referenced model fails (e.g. a masked test object that
defines initialization code in the mask). Previously, this
could lead to an easy-to-overlook failure when trying to
set up the coverage.
#9076: This fix eliminates the Test Data Viewer crashing when
trying to display expected output signals in case the
corresponding data file is empty.
#9088: This fix resolves a testbed simulation failure due to a
version mismatch of referenced models within the testbed.
Now the model reference block within the testbed is
automatically refreshed during testbed generation.MTest v 7.4.0 (April 2021)
- Batch Regeneration of Test Sequences from MARS Requirements
Similar to the batch regeneration of assessments the user is now
able to regenerate test sequences from MARS requirements within
the batch test. For this purpose, the new option
'Regenerate & Import Test Sequences' is available within the Batch
Test GUI.
Please note: During batch regeneration of already existing
generated test groups, the corresponding MTCD files (*.mtcd) are
backed up in the corresponding test object directory of the editor
workspace (i.e. <TESTPROJECTDIR>/editorWorkspace/...
<MODELNAME_TESTOBJECTNAME>/*.mtcd_backup_YYYY-MMM-DD_HH-MM-SS). - User Rationales for Model Coverage Filtering (#8838)
Within the Coverage Filter Configuration, the user can now provide
their own comments to justify ignoring the selected subsystem.
These annotations are also included in the corresponding
Simulink Coverage Report. For this purpose, the field 'Rationale'
has been added to the structure containing the ignore list.
Please refer to the example below: CovFilterConfig.IgnoreList = [ ...
struct('Path', '<SUBSYSTEM_A/<SUBSYSTEM_B>' ...
,'SID' , ''...
,'Rationale' , 'This is excluded because of ...'...
); ...
]; - Selection of Subsystems within Referenced Models for Coverage Filtering
Subsystems of a test object that are located within model
references can now be selected via the selection GUI.
In order to be able to select these subsystems, it is necessary
that the testbed is built while resolving the model references.
For this purpose, it must be ensured that the corresponding
project parameters 'TargetLinkResolveRefModels' and
'EmbeddedCoderResolveRefModels' are each set to 1. - Ignoring of Model Block Instances for Model Coverage Filtering (#8838)
In addition to the existing possibility of excluding entire
subsystems from the structural coverage calculation, it is now
possible to exclude individual block instances from the model
coverage calculation. Please see the example below: CovFilterConfig.IgnoreList = [ ...
struct('Path', '<SUBSYSTEM_A/<SUBSYSTEM_B>/<BLOCK_X>' ...
,'SID' , ''...
,'Rationale' , 'This is excluded because of ...'...
); ...
]; Please note: This type of exclusion is only available for the
model coverage. If your use case calls for both model and code
coverage you are strongly advised to only ignore subsystems.
If block instances - other than subsystems - are ignored you are
advised to disable the code coverage filters by setting your
configuration parameters 'TLCodeCoverageFilterEnabled' and
'CTCCodeCoverageFilterEnabled' respectively to 0. - Enumeration Members Available for Use in MTCD
(#6417, #7875, #8609, #8611)
With this release, enumerations can be used within MTCD test
descriptions. For this purpose, the required
enumerations and their members must be declared in the interface
file. Here is an example: enum Color {
Red
Green
Blue
} Please note: The declaration of the enumerations in the
corresponding *.io file must be done manually for now. Automatic
retrieval of the enumerations used by the test object will be
available in future MTest releases. - New Option for Handling Input Data Type Conversion within the TestBed
We introduced a new configuration option that controls the
settings of the data type conversion blocks in the generated
testbed.
For this purpose we introduced the parameter
InheritInputDataTypes. It can have the following values: 0: Inactive. (Input data type conversion is set to the data
type according to interface analysis)
1: Active. (Input data type conversion is set to
'Inherit: Inherit via back propagation'.
The default value is 0. Please note: While this option is active it may lead to signals
with under-specified data types - depending on the test object. In
this case Simulink will display specific warnings. - Discontinued Support for MATLAB R2011b to R2013a
This release officially does not support MATLAB R2013a to
version R2011b. From now on, MTest officially supports MATLAB
R2013b-R2020b. - Further Bug Fixes
#8920: This fix ensures that the testbed is compilable during its
generation. In the past, this could lead to errors with
TargetLink test objects that required a rescan of the buses.
#8928: With this fix the access to parameters of type
Simulink.Parameter in MARS has been simplified. These can
now be queried directly, i.e. access via
MySimulinkParameter.Value is no longer necessary.
#8943: This change has made the documentation of the project or
test authors more consistent. The catalogs now uniformly
list the project author in their respective headers, while
the headers of the test reports, on the other hand, show
the respective test authors.MTest v 7.3.2 (March 2021)
- Data Logging for Testbeds with Test Objects Integrated as
Referenced Models
With the introduction of Simulink Code Coverage support for
Embedded Coder models, we had to adapt the test bed generation
accordingly (see also Release Notes for MTest v 7 3.0). The test
objects are integrated here via reference models. With the release
of MTest v 7.3.0, test beds built in this way could cause errors
when saving and applying user-created data logging configurations.
These errors are now fixed with this update release. - Ignoring Model Coverage for Testbeds with Test Objects Integrated as
Referenced Models
For testbeds integrating the test object as a referenced model
(see above) the user is now able to select the subsystems which
shall be ignored during coverage measurement. Also, already
existing coverage ignore configurations can now be applied to
those testbeds. - Improved Batch Generation of Assessments from MARS Requirements
With this update release we facilitate the generation of
assessments from MARS requirements when the batch test is set up
programmatically. - Further Bug Fixes
#8565: Solves the problem that configuration files were mistakenly
detected when they are located in the MATLAB path, but
not in the project tree, e.g. in the model folder.
#8781: This fix ensures the proper detection of data types on
TargetLink bus elements whose destination block is a
TargetLink AUTOSAR block.
#8816: With this fix we make sure that referenced configuration
sets are resolved properly. In the past, the testbed
generation caused an error leading to testbeds that cannot
be simulated.
#8837: This fixes an error during import of MTCD test sequences
that do not contain any assignment of values to signals.
#8841: This fix prevents the MiL simulation being aborted in case
Simulink.Parameter are stored in the Simulink Data
Dictionary. In the past an error occured due to the fact
the data was defined in the both the Data Dictionary and
the base workspace.
#8852: This fix enables code generation for Embedded Coder models
in MATLAB R2020b.
#8853: With this fix we make sure that the Simulink Data
Dictionary receives an update when the user loads test case
data via the menu option "Execution -> Load Data for Debug
Simulation"MTest v 7.3.1.Linux (February 2021)
- First Prototype for Use of MTest in Linux CI Environments
In the context of CI applications, MTest can now also be used on
Linux operating systems, especially for the execution of CI batch
runs. The following functional limitations should be noted: - Testing of TargetLink models is not supported.
- Project paths must not contain backslashes.
- The older Excel format *.xls is not supported. Prior to
import, please convert your requirement and/or MTCD test
specification documents, originally saved as *.xls files,
into *.xlsx files.
- Collection of code coverage data with Testwell CTC++ is
not supported. However, code coverage with Mathworks'
Simulink Coverage Toolbox is working.
- The application of custom data logging configurations
(originating from modified testbeds) cannot be performed.
- Excel documents cannot be opened from within MTest.
- The MTest Specification Editor is not part of the prototype
release and therefore cannot be used.
- Please note that it might be possible that currently limited
functionality, e.g. support of *.xls files, may lead to
MATLAB errors due to missing interceptions in all misuse
cases.
- The default build directory ('C:\work\MTest_Batch_Projects\')
was not adapted in MTest_Demo_Build_Autopilot.m (Line 114).We encourage all Linux users to report any further issues.
MTest v 7.3.0 (December 2020)
- Embedded Coder Code Coverage
Users of embedded coder models can now also collect their code
coverage using the Simulink Coverage Toolbox.
For this purpose, the creation of the testbed has been changed.
The selected test object is converted into a referenced model and
automatically inserted into the testbed. In order to be able to
use the Simulink Coverage Toolbox, the corresponding project
configuration must be adapted. The parameter
CovEmbeddedCoderCodeCoverage was introduced for this purpose.
The parameter can have the following values: 0: Testwell CTC++
1: Simulink Coverage
The default value is 0. The following limitations remain in effect:
- Simulink Coverage is only available for MATLAB R2018b
and newer
- Coverage ignore mechanism is not supported - Batch Creation of Assessments
In this new version it is possible for the user to generate
assessments from MARS requirements within the batch test. For this
purpose a new Batch Action (Regenerate Assessments) is available
within the Batch Test GUI.
In this context, the GUI layout of the available batch actions has
also been redesigned. In particular, the options for updating
(or regenerating) (MARS-based) test elements are now arranged more
clearly.
Please note: Before executing the batch generation of
assessments from MARS requirements, the corresponding storage
directories of the assessment files are deleted each time. - Automatic Read Out of Signal Characteristics of Bus Elements (#8535)
The test case generation has been improved in that the signal
characteristics of bus elements are automatically read out from
the test object input interface and from signal attribute
definitions defined higher in the model hierarchy (if the model is
compilable). The user no longer has to enter this information
manually in the <TESTOBJECTNAME>_interface.io file. - Further Bug Fixes
#8472: Previously, function calls were being misinterpreted as
data type, resulting in them being used in the data type
conversion block. This lead to errors when trying to
simulate the testbed. This has been fixed now.
#8566: This solves the issue of MARS assessments not being counted
and displayed in the main GUI and evaluation configuration
GUI. However, they have always been evaluated.
#8591: With this fix we solved the problem of losing the filter
configuration when leaving the changed testbed unsaved,
and the user (or MATLAB) closes the testbed. Now we save
the testbed after changing the coverage filter settings.
#8652: This fix supports the specification of relative paths
within the testbed's TargetLink Data Dictionary using '.\'
as well.
#8658: This fixes an issue of logged signals not being available
to the assessments. This could have occurred in the batch
mode in case both the MiL and SiL simulation were activated
at the same time.MTest v 7.2.0 (September 2020)
- ReqIF Export of MARS Requirements
Formalized requirements created with MARS can be exported to
*.reqif file format now. The export file is created automatically
when saving the *.mars file in the MTest Specification Editor.
The file is stored alongside your *.mars file under:
\<testProject>\editorWorkspace\<testModel>_<testObject>\...
<testModel>_<testObject>.reqif
Info: As of yet, the specifications of defines are not exported to
the *.reqif file. This will be done in a future release. - Testbed Generation Supports Resolving of Referenced Subsystems
(#7923, #7957)
With this release we support the resolving of referenced
subsystems (as introduced in R2019b) during testbed generation.
Referenced subsystems are handled in analogy to referenced models,
i.e. they are converted to subsystems. This only affects the
testbed, there will be no changes to the source model. - New Default Handling of Library Links and Model/Subsystem References
When generating a testbed all links to libraries and referenced
models/subsystems are resolved by default. This only affects the
testbed, there will be no changes to the source model.
The respective elements are converted to subsystems to ensure an
invariant testbed. However, the behavior can be changed by means
of the respective project configuration parameters. These are: - TargetLinkResolveRefModels - Valid options are:
0: The testbed will refer to the original referenced
TargetLink models
1: All references to TargetLink models will be resolved
(i.e. converted to subsystems)
The default value is 1. - EmbeddedCoderResolveRefModels - Valid options are:
0: The testbed will refer to the original reference
Simulink/Embedded Coder models or subsystems
1: All references to Simulink/Embedded Coder models or
subsystems will be resolved (i.e. converted to
subsystems)
The default value is 1. - TestbedBreakLinks - Valid options are:
0: Original links to libraries will remain in the testbed
1: Links to libraies will be broken (i.e. converted to
subsystems)
The default value is 1. - Improved Synchronization of Workspace and TargetLink DD (#7110, #7988)
In order to vary parameters that are defined as Data Dictionary
variables (ddv) MTest now ensured ddv parameters are synchronized
with your test (sequence) data prior to each simulation. Previously,
this was only handled in batch mode.
With this release it is also ensured that the synchronization also
takes place when simulating the test sequence manually via the main
GUI. - Improved Handling of Coverage Data (#8060, #8477)
Coverage data handling is improved such that MTest will clear
possibly outdated data from the project. This ensures that the
test sequence and coverage reports always contain up-to-date data.
This ensures that no "old" coverage data is present, for instance,
in case a test sequence re-simulation has failed. You are advised
to rerun your TargetLink Code Coverage simulations when updating
to MTest 7.2.
Info: Already determined code coverage data will not be deleted
after the regeneration of code. - Display Effective Interface Data in Test Sequence Report (#8335)
The sections "Test Input" and "Test Output" of the Test Sequence
Report show the respective signal data as specified in the
Effective Test Interface. - Requirements Traceability for Generated Test Sequences (#8109, #8357)
Test sequences - generated from MARS formalizations - do also
link to external requirements now. This is particularly beneficial
for users that refer to external requirements as the base for
requirements traceability. Please note that the requirement link
of the respective MARS formalization is used. - Ignore Coverage of Subsystems Within Libraries (#8256)
In case subsystems to-be-excluded from coverage measurement reside
within a library, those subsystems can now be ignored when using
the option. - Source for Signals Names (#8500)
Now, the user has the possibility of specifying the source for
naming the testbed's input and output signals. The selection is
made using the project configuration parameter
"SourceForSignalNaming". Valid options are:
0: signal names are used
1: port names are used
The default value is 0.
Please note: For interface bus signals it may be the case that we
still have to fall back on using the respective signal names of
the bus elements. - Further Bug Fixes
#6900: We ensured a unique namespace of MTest-internal functions
(e.g. findfiles.m renamed to MTu_findfiles.m) in order to
avoid name clashes between MTest and customer (utility)
functions.
#8036: Fixes the issue that an MTest_Configuration.m file is
created in the current folder while changing from the
current project to "No project selected...".
Note: The configuration was empty, thus not affecting
the project.
#7235: Fixes the issue of not opening the proper Data Dictionary
(i.e of the source model) while adding a new test object to
the project.
#7920: This solves the issue of MTest not being able to open the
Model Coverage Dialog when using MATLAB R2019a and newer.
#8123: This fix solves the issue of not measuring the MCDC
coverage when the option "ReGenerate TestBeds" is enabled
during Batch Test. Instead, a default setting (Condition,
and Decision Coverage) was used.
#8146: Fixes an issue of generated assessment whose the trigger
condition contains a duration expressed by means of an
equation. Now, a proper parenthesization is ensured.
#8196: This fix ensures the reset of the TargetLink Simulation
mode when changing the test project.
#8235: Fixes the issue of testbeds not properly reflecting the
structure of buses defined in the TargetLink Data
Dictionary.
#8285: This fix will prevent that expected output definitions
during measurement data import will be copied into output
data files. Under certain circumstances, this could have
led to confusing evaluation results. This safeguards that
expected output definitions will have an impact on other
evaluation methods.
#8295: Solves the error of vector length mismatch during MiL
simulation. This issue could have occured for MiL
simulations in fast restart mode while the Simulink
configuration parameter LimitDataPoints is active. Now, we
try to deactivate it.
#8296: Solves the "s-function not found" issue when running a
Batch Tun with a TargetLink model, using the Fast Restart
option. In case no code was generated before, this
particular combination of actions could result in the code
generation not triggering during the batch run.
#8326: Fixes the issue where the MTCD specification document is
repeatedly imported after each batch action while
simulating the testbed in fast restart mode.
#8334: This fix catches the error that occurs while trying to
open the model coverage dialog via the MTest GUI in case
the Simulink Verification and Validation Toolbox or
Simulink Coverage Toolbox are installed on the user's
system but do not feature a valid license for those
toolboxes.
#8339: Fixes an issue when generating assessments based on formal
requirements that refer to parameters that are of integer
data type.
#8358: Solves the issue of global simulation parameter
(e.g. Variable-Step Solver) not being applied to Embedded
Coder models during Batch Test.
#8384: Fixes an issue that lead to a crash while trying to using
the MARS expression 'the system starts' within a define.
#8393: Fixes an issue where, in rare cases, the TargetLink code
coverage data contained outdated datasets. We make sure
that there is only one data set of TargetLink code coverage
per test sequence. Old data, e.g. from previous runs, is
not considered and thus it is made sure that the code
coverage data originates from the latest test sequence
execution.
#8436 This fixes an issue where buses have not been resolved
properly in order to make the individual bus elements (i.e.
signals) available in the *_interface.io file.
#8451: This fix ensures that the testbed generation includes the
automatic rescan of the testbed's TargetLink Bus Ports.
Please note: To be able to rescan the ports, the testbed
needs to be capable of being updated. Therefore, the
testbed needs test data in order to be compilable. You may
get an error message regarding a failed bus port update
upon initial testbed generation. The testbed is still build
correct whatsoever. You are advised to regenerate the
testbeds after you have imported a test sequence.
#8456: See #8393.
#8472: Fixes an issue with the testbeds of a triggered subsystems.
Here, function calls were misinterpreted as data types and
thus trigger signals mistakenly underwent a data type
conversion.
#8490: Improvement to the file header descriptions of
MTest_BatchProject.m und MTest_ExecuteBatchTest.m.
#8493: This fix reacts to warnings during testbed generation in
MATLAB 2020a due to deprecated block parameters of the
Data Type Conversion block.
#8510: Previously, when the code generation failed, there were
error messages claiming an invalid Simulink object within
the testbed, that occurred in all MTest actions thereafter.
This fix ensures the validity of the testbed even after a
failed code generation.
#8536: See #8436.
#8548: With this fix we ensure that the TargetLink simulation mode
is always set back to MiL when regenerating the testbed.MTest v 7.1.0 (June 2020)
- Import of External Requirements Using the *.reqif Format
With this release, MTest enables you to also import your
requirements from *.reqif files (Requirements Interchange Format).
The import is fully configurable. All import tags can be set in
order to read the respective requirement attributes (e.g. ID,
Description, etc.) according to the custom format of your
requirements document. Also, by defining the specification type
the import can be tailored to import only specific requirements.
In this release, the import of attributes is limited to
string data type (i.e. attributes of type XHTML are not supported,
yet). - Graphical User Interface for Configuration of Requirements Import
Now, the import of your requirement documents (*.xlsx and *.reqif)
is fully supported by a graphical user interface. You can access
the interface from the MTest menu bar via
"Requirements / Configure Requirements Import" or via the button
"Configure" in the "Requirements" panel.
Here, you can easily specify the import settings, i.e. the
elements of your requirements document that shall be carried over
to MTest. Also, the setting of filter rules for the automatic
determination of the requirements' testability was improved to be
more intuitive and user friendly. - Further Improvement of Test Case Generation
This new release includes some further improvements to the
requirements-based test case generation. These are:
- The ranges of your interface signals ([Min Max]) are read from
the model directly, so that your already implemented signal
ranges are carried over to the MTest Specification Editor
automatically (upon initial creation of the *_interface.io file).
Please note: The Min/Max values are not read from TargetLink
ports.
- Boolean signals are now handled automatically. You do not have
to specify signal range and resolution for this signal type.
- With respect to the signal data type, the resolution is set
automatically (i.e.
- Boolean, integer -> resolution = 1;
- single, double, fixdt -> resolution = 0.1
However, the user can specify a custom resolution value within
the *_interface.io file.
- The internal test case generation algorithm was made more
robust in order to prevent any memory overflows during solution
computation. - Extension of Project Configuration
The evaluation configuration (assessment, signal comparison,
expected output) is now integrated into the project configuration.
On the one hand the user benefits from the convenience of
specifying the evaluation configuration for a given test level
(test project, test model, test object, test group, test
sequence). On the other hand the evaluation configuration can be
defined within a central, external project configuration and thus
can be distributed to different MTest projects by means of
referencing it. In addition, it is now possible to specify that a
given test project level (e.g. test group) shall inherit its
evaluation configuration from a level that is higher up in the
project tree (e.g. test object or external). Furthermore, the
evaluation configuration GUI was enhanced to better browse the
test project tree. Now, the user has the option to display all
test groups and/or test sequences.
Upon initial opening of an MTest project created with MTest 7.0 or
older, the migration of the previous evaluation configuration is
carried out automatically. - Exclusion of Selected Signals from Regression Test
With this release, the user can now select which signals shall be
excluded from a regression test. The evaluation result for those
signals is rendered to 'Not Triggered.' You can access the
interface from the MTest menu bar via
"Evaluation / Manage Reference Data Set." - Improved Handling of TargetLink Parameters
The interface analysis was enhanced to recognize TargetLink DDV
parameters. Now, they are read from the model and thus are
available for use in MARS. - License Injection Via MES Jenkins Plugin
Now, CI users of MTest do not have to perform the license
configuration on each slave-machine of the CI setup individually
(i.e. MES Jenkins Plugin and multiple instances of MTest). Once
the license of the MES Jenkins Plugin is configured properly, this
information is passed on to MTest and is prepended to the license
configuration of MTest. - Elementary MATLAB Constants Available in MARS (#8107)
It is now possible to use elementary MATLAB constants
(e.g. pi, eps, i, inf) within MARS requirements. - Further Bug Fixes
#7482: Fixes an issue when the source model utilizes the
Data Import/Export configuration parameter
'Single Simulation Output.Â’ Previously the simulation data
could not be retrieved when this option was active. This
problem has now been resolved.The option is deactivated for
the simulation.
#7894: This fix provides drastically accelerated test sequence
import by optimizing the analysis for logged signals.
#7902: This fix enables the user to log signals if the test object
is a top-level Stateflow chart.
#7922: This fixes an issue with a runtime exception being thrown
during Test Case Generation (TCG) when respective MARS
requirements were using enumeration assignments. Those
requirements are now skipped during TCG.
#7926: In the MTest Specification Editor "~=" and "not equal" are
now part of the Content Assist.
#7940: This fixes an issue during interface analysis in case a
test object, which is a referenced model, has bus inputs.
#7944: The Simulink Data Dictionaries referenced by the currently
selected test object are now closed upon selection of
another testbed. This prevents issues while compiling the
model due to model and testbed data dictionary being open
at the same time.
#7956: This fixes an issue of a failing interface analysis in case
(sub-)bus objects have been defined within a Simulink Data
Dictionary.
#7971: During interface analysis, MTest also collects all
parameters (i.e. workspace variables) of test objects that
resemble referenced models. With this fix those parameters
are now available for use with MTCD and/or MARS.
#7986: This fix enables MTest to recognize signals logged with
TargetLink even if the test object (i.e. the selected
TargetLink subsystem) is set to SiL mode.
#8002: This fixes an issue where a Simulink Data Dictionary is
being opened once the testbed has been saved. This could
have lead to a simulation crash caused by conflicting
(multiple) definitions of data due to multiple loaded data
dictionaries. See also bug fix #7944.
#8004: This fixes an issue where a testbed was not generated in
case an input bus contains elements that are of type
enumeration only.
#8041: In case of multiple bus object instances originating
from a Stateflow chart the testbed generation now builds
bus selectors with the names from the model and not ones
that were renamed by MTest.
#8053: Fixes an issue related to test sequences that only set
parameters, and not any signals, that were
only one time step long, ignoring the set duration.
#8129: This fix ensures naming consistency when storing logged
signals data and displaying them in the *_interface.io
file. In the past, this could have led to errors handling
logged signals whose name string contained inadmissible
characters.
#8123: Solves an issue where the coverage settings in the testbed
were reset after regeneration. This fix is especially
helpful in batch mode.
#8151: This fixes an issue during test bed generation in case
buses are terminated within a test object.
#8188: This fix solves an issue regarding a runtime error when
MATLAB is not able to read an MTCD Excel document during
the test sequence import.
#8196: This fix ensures that the TargetLink simulation
mode is reset correctly, e.g. when changing the test
project or test object.MTest v 7.0.1 (February 2020)
- Performance Improvements of Test Sequence Imports (#7648)
Crucial data gathering procedures have been reworked so that
they do not have to be invoked for individual test sequence
imports. Thus providing a significant speedup. For instance,
those procedures concern:
- Identification of test object parameters
- Retrieval of logged signals
- Compilation of parent model - Extended interface analysis of Simulink Bus Objects (#7111, #7377)
With this update, MTest is able to create testbeds for test
objects containing signals originating from multiple instances of
(nested) Simulink Bus Objects. To provide unique signal names,
processable within MTest, a new naming pattern was introduced for
those bus elements. The pattern reads:
<busname>_<nestedbusname>_<*>_<buselementname>
Per default this naming pattern is inactive. However, the user
has the possibility to activate this option by means of changing
the testbed configuration parameter config.BusElementRenaming
(see MTest menu: "Organization / Manage Project Configuration").
Despite of the naming pattern being inactive, MTest detects the
necessity of activation during interface analysis. In that case,
the user has the possibility to let MTest automatically adapt
the configuration accordingly. - Recursive Breaking of Links to Libraries and Referenced Models (#7627)
MTest supports the breaking of links during the testbed generation.
This works especially well when resolving model references, in
particular when those references are positioned within libraries.
Both features safeguard the invariance of the testbed throughout
the testing process and give the tester flexibility for setting up
the testbed according to the requirements of the particular test
project. To activate this feature, a dedicated option within the
configuration management is available for test object and test
project level: config.TestbedBreakLinks. - Access of Elements of Vector Parameters in MTCD (#7672)
The syntax of the Model Test Case Design language was extended to
enable access to individual elements of vector parameters for use
within the test sequence specification, for example:
p myParamElement = myParamVector(12);
Please note: There is no index verification, so the user must
ensure that index does not exceed the number of array elements. - Row-/Column-Wise Variation of Matrix Parameters in MTCD (#7207)
With this version MTCD supports the variation of entire rows
and/or columns of matrix parameters. See below for an example: myMatrixParam = [1 2 3; 4 5 6; 7 8 9];%
Initialization
p myMatrixParam(2,:) = [10 11 12];
// or
p myMatrixParam(:,3) = [13 14 15]; Please note: There is no index verification, so the user must
ensure that index does not exceed the number of array elements. - Execution of MTest and MXAM at the Same Time
There is a demo script available (\demo\MTest_MXAM_SideBySide.m)
demonstrating the proper path initialization prior to running
MTest and MXAM, concurrently. Please also refer to Chapter 2.5 of
the user guide for a detailed rundown.- Further Bug Fixes #7231: Fixes an issue regarding TargetLink mux blocks where the signal properties have not been transferred properly to the testbed. #7449: MTest supports the white-listing of testbed subsystems and referenced models by utilizing the settings defined in the Coverage pane of the testbed model's Configuration Parameters. #7484: Enables the generation of testbeds for Stateflow Chart test objects outputting buses. #7535: Fixes an issue while generating a testbed based on a parent model utilizing a variable-step solver. In this regard, we ensured the proper handling of the sample time constraint settings. #7628: Fixes an issue while resolving model references within the testbed. In case, the referenced model utilizes a variable step solver the sample time of new subsystem is set to inherited #7649: Fixes an issue regarding the coverage ignore-list allocation of TargetLink subsystems with the TargetLink simulation frame not being on the top level of the test object. #7651: Fixes an issue where logged signals showing a custom signal name have not been stored properly. #7659: Fixes an issue where the ports of TargetLink subsystems - allocated to the coverage ignore-list - have not been enhanced properly (see also #7649) #7740: Fixes an issue where the conversion of time units was not executed for the MARS expression <change with a slope>. This could have provoked errors during assessment generation.
MTest v 7.0.0 (December 2019)
- New Command for Starting MTest
Now, MTest can be started using the command >> mtest. This opens
the main GUI and loads the most recent project. To start MTest
without any project being loaded, use the command >> mtest('none'). - New Configuration Management
With the new release of MTest, we now provide a method for
configuring your test projects more conveniently. You can define
configurations on each available test level, i.e. test
project, test model, test object, test group, and test
sequence, respectively.
Furthermore, you are given the opportunity to refer to an external
configuration so that test project setups can be standardized to
provide a global/company-wide configuration. However,
configuration parameters can be overwritten through specific test
level configurations.
This new management supersedes configurations that - in the past -
have been specified within the MTest_UserPrefGUI.m and the
MTc_RCToptProjectSettings.m, respectively.
In addition, the configuration parameters are also grouped within
categories so that options can be found more easily.
You can access the configuration management from the menu bar of
MTest main GUI via "Organization / Manage Project Configuration".
The first time you open the new MTest it takes care of migrating
previous configurations. - Integration of MARS Requirements into the Test Documentation
Now, requirements that have been formalized with MARS are
represented in the requirements catalog. An overview of the
particular MARS items related to the requirement is now provided.
In conjunction with the new configuration management, you now
define the source of requirements that shall act as the reference
for traceability and metric computation. Available options are:
- External requirements (e.g. as imported via Excel)
- MARS formalizations (e.g. as created with the MTest
Specification Editor).
Therefore, the requirement catalog was enhanced to support
documentation for users that work with MARS formalizations only.
In order to generate this updated catalog representation for
existing test project you are advised to save your existing MARS
document again from within the MTest Specification Editor. - Change Impact Analysis Available for MARS Requirements
With regard to requirement changes, MTest can now automatically
update the attributes "description" and "status" of the MARS
formalizations that link to a changed external requirement.
For MARS formalizations affected by a change, its "description"
attribute will automatically reflect the new requirement
description. Similarly, the "status" attribute changes to
'reqchanged'.- Online Parameter Modification for SiL Tests of TargetLink Models TargetLink users can now speed up SiL testing by taking advantage of online parameter modifications, which render the regeneration of code unnecessary in case of parameter variations within test sequences, i.e. code is generated only once per test object. The feature is enabled by activating the respective checkbox in the MTest batch test GUI. Please note: This feature is only available to users of TargetLink 4.0 and newer.
- Test Case Generation by Variation of Synchronized Lists
Users of the test case variation feature can now mark lists of
variation points that are kept synchronous during test case
generation. Such lists will not be considered for a full
combination, instead those value groups enter the combinatorics as
single-entries, i.e. a variation is comprised of the combination
of variation list members sharing the same element index. - Further Enhancements of MARS
Now, MARS supports unrestricted use of nested "defines", i.e.
event defines are available to state define and vice versa. Nested
defines do work recursively, as well.
The status tags 'new' and 'proposed' became deprecated. A quick
fix solution is available in the MTest Specification Editor. The
following mapping is used to replace the old values in order to
ensure consistency to MTCD documents and assessment files:
- 'new' -> 'created'
- 'proposed' -> 'described'
Also, MARS users can now make use of value tolerances within the
expression 'change with a slope [+- myValue]'.- Adaptions Assessment Utility Functions The following assessment utility functions are now capable of handling vectors containing NaN values: - MTa_differentiate - MTa_isFall - MTa_isRaise
- Bug Fixes
#7313: Fixes the issue of the model coverage aggregation slowing
down the simulations within the batch test. Now, the model
coverage is aggregated after the execution of the last test
sequence of the batch run. This even works when not all
test sequences of a test group have been selected.
Note: Aggregation is not performed if the batch test is
aborted by the user by clicking the "Stop" button.
#7339: Fixes an issue where the testbed's Simulink Data Dictionary
could not be found although it was present and linked to
the testbed model file.
#7407: Fixes a bug where names of logged signals containing
blanks, new lines and/or special characters could not be
displayed properly in the MTest Specification Editor. In
the past this could have yielded syntax errors within *.io
files
#7431: Fixes the issue of not showing a blank "Project Settings"
window when setting up a new project while project is
loaded in MTest. Instead of empty attribute fields, those
fields showed the content of the currently loaded project.
This caused confusion with users thinking they change the
current project settings.
#7463: Fixes the issue where the source model was compiled during
each test sequence import. This significantly slowed down
the entire import process.
#7464: See #7463.
#7487: Fixes an issue where parameters of type Enumeration have
erroneously been casted to double.
#7592: Fixes an issue where the simulation of test beds - using a
variable-step solver - in the base workspace yields
a time vector unequal to the length of the signal output
vector(s).
MTest v 6.4 (September 2019)
- (Alpha) Functional Test Case Generation Based on MARS Requirements
With the new release of MTest, we now provide a method for deriving
test cases on the basis of formalized requirements created with the
MTest Assessable Requirements Syntax (MARS). For given types of MARS
requirements, the user is now able to automatically generate test
cases (incl. test vectors) that will trigger the model behavior as
specified by the requirement.
This approach accelerates the creation of test cases. Moreover, the
manual effort in composing a test suite is reduced, since both ends
of testing - from simulation to evaluation - are now covered by MARS.
Currently, this feature has alpha status which means that our users
are highly encouraged to provide us with their feedback and ideas.- New Representation of the Test Project Protocol The representation of the Test Project Protocol - MTest's feature for checking the integrity of your test projects - has been completely restructured so that test object specific issues can be identified faster. For this purpose, the failed checks for correctness, completeness, consistency, as well as the suggestions for any follow-ups are listed for each test object individually. This aids navigating, checking, and counteracting issues you might encounter within your test projects.
- Improved User Guide
MTest's user guide is now available in HTML format allowing
convenient use of document navigation and search (including
search term highlighting), as well as bookmarking. A PDF print
version of the user guide is still available as well, see
'<MTESTROOT>\doc\userguide\MTestUserGuide.pdf' - (Beta) Test Case Generation by Variation in MTCD
The demo project - that can be set up automatically - now
contains a sample test group (Autopilot_Mode_Logic\TestGroup003)
that showcases the use of variation for deriving test sequences.- Test Object Selection When adding a test object to the test project, the user is now able to select the current system (as retrieved with GCS) as a new test object. This is a fast and convenient way to include a test object without potentially navigating extensive subsystem structures.
- Evaluation Configuration
The user can now specify whether manually
created assessments and/or assessments generated with MARS should
be considered for automatic test evaluation. This decouples
both assessment types and eliminates the need to specify
attributes for manual assessments, in case only MARS assessments
were used.
- Useful Links in Main Menu Quick access to useful websites is provided from the help section of the main GUI menu. Now, the user can directly open the following webpages: MES homepage, MES Webinar Series, and MTest Support.
- Bug Fixes
#7022: Fixed issues during testbed creation for test object
utilizing bus objects and custom data types. This fixes
a bug that occurred in MTest v6.2.1.
#7108: Fixed issues with logged signals holding signal names that
contain blanks, e.g. problems with *.io files and
problems with tolerance configuration.
#7147: Fixed issues with MTest batch project in the context of
CI usage, where errors regarding the selection of
TargetLink code coverage selection occured.
This fixes a bug that occurred in MTest v6.2.1 and v6.3.
NOTE: If you encountered this issue, updating your existing
batch selection mat files is required. To do so, open the
batch test GUI, select your desired batch actions, and simply
save your selection.
#7155: Fixed an issue with a confusing command window output when
an entire test group was selected in the batch GUI, in
conjunction with an active re-import option. In the past, a
single sequence re-import was displayed even though it
was skipped after successful re-import of the test group.
This fixes a bug that occurred in MTest v5.0.
#7168: Fixed an issue when analyzing the data type for a bus signal.
In some instances, the data type was resolved to inherit
instead of using the assigned bus object. This might lead to
issues for data type conversions, when a specific data type is
required. This fixes a bug that occurred in MTest v6.2.1.
MTest v 6.3 (July 2019)
- (Beta) Test case generation by variation in MTCD Test sequences can be generated using logical test cases in MTCD in the MTest specification editor. The logical test cases can contain any number of variation points. Each variation point is defined by a variation list. During test sequence import concrete test sequences using the defined variations are derived from the logical test cases. The number of generated test sequences depends on the defined variation points and the number of elements in each variation list used. Currently a full variation algorithm is used to generate test sequences.
- (Beta) Logged signals utilization in signal comparisons Signal comparisons were extended to logged/local signals (in addition to output signals). Logged signals can be used in regression, back-to-back and expected output comparisons. To activate logged signal comparisons for respective evaluations, please enable this feature in MTest_UserPrefGUI by the options: PropGUIDefault.evaluation.comparelocalsignals.Regression = 1; (respective BackToBack = 1; and ExpectedOutput = 1;)
Definition of tolerances is supported for logged signals as well.
You find the logged signal names in the tolerance option file.
These logged signal can be recognized by the extension "(logged)".
The results of the comparisons are reported in the evaluation
section of the test sequence report, in test sequence catalog,
and in signal comparison catalog.- Ignore specific signals in signal comparisons by INF tolerance To exclude/ignore some signals from signal comparisons, just define a tolerance of Inf for these signals. If any of the value tolerances or the time tolerance is Inf, this signal is evaluated to NOT ACTIVE. This result is reported in the respective reports and catalogs. It is visible by design, but will vanish during result aggregation. This excluding configuration applies to all test sequences of a test object for the respective signals.
- Improvements in Interface Analysis When a BusCreator is found the Interface Analysis takes all the information from the BusCreator-Struct on output and input side. On input side there was an additional effective interface analysis, which could produce incorrect interface structures. (#6888)
- Support for configuration references in SLDD Configuration references in SLDD are handled by MTest now. It is ensured, that SLDD of testbed and model are not open at the same time, when a model update/compile is performed by MTest. (#6906)
- MARS: 'inbetween' with open and half-open intervals (include, exclude) For the inbetween definition it is now possible to define the exclusion of the boundaries by using the keyword exclude. The keyword include was also added but it defines the default behavior. (#6857)
- Measurement data initialization in MTCD in expected order For the import of measurement data using MTCD, the order of initialization of signals was improved. Definitions in the test sequence have precedence compared to definitions in the test group. It is still possible to "overwrite" signals of the measurement data (or add missing ones) by defining them in the initialization or action field of the respective test sequence.
- Master-slave support for MTest step in MES Jenkins PlugIn The MTest step of the MES Jenkins PlugIn supports Master-slave now. An MTest-CI-License is necessary to use this feature.
MTest v 6.21 (May 2019)
- Inclusion of referenced models in SLDD scenarios In EC (and SL) models using SL Data Dictionary there exists now an option to include the contents of referenced models into the test object in test bed. (The same mechanism already existed for TL models.) In this way the test of models with referenced models using each an own SLDD is possible (without effecting the original models or SLDDs). (#6741) To enable this inclusion, please activate in MTest_UserPrefGUI the following option to 1 (default is 0): PropGUIDefault.execution.ecdisablerefmodels4testbed = 1
- Enhanced interface analysis The handling of bus objects was extended to handle vector signals (Nx1 and 1xN arrays). This extended handling prevents the sometimes erroneous handling of bus objects and busses introduced in 6.2. (#6782)
- Read baseline of requirements from requirements document The import of requirements now supports the reading of the baseline of the requirements from (Excel) file. Just put the text of the baseline in the description column in a row of type 'Baseline'. In this way the import of requirements can be done in a complete batch manner. (#6786)
- Workflow enhancements
- In 6.2 it was possible, that the TL code coverage was not
correctly aggregated (when running MiL and SiL in batch mode
together in the same run, thus interchanging MiL and SiL sims
of the test sequences of one test object). There was no problem,
when MiL or SiL were run in separate batch runs. Now the
internal setting of the TL sim mode is correct again. (#6833)
- The numbering of the test sequences and assessments in the
Assessment Matrix Catalog matrices reflects now the test
seq number and not the index of the test sequence (was only
visible when deleting some of the first test sequences). (#6751)
MTest v 6.2 (March 2019)
- Speed-up of batch MiL simulation by making use of Simulink Fast Restart The batch test now allows the use of Simulink's Fast Restart feature for executing of MiL simulations of Simulink and Embedded Coder models. The user can activate this option via a dedicated check box in the Batch Test GUI. The Fast Restart is only available in the following versions: - MATLAB R2015b and newer (#6871) - TargetLink 4.4 Furthermore, the Fast Restart only allows to simulate test sequences sharing the same step size. In case of varying step sizes the test sequences selected for batch execution will be simulated using the step size of the first test sequence.
- Speed-up of batch SiL simulations by using tunable parameters
User of Embedded Coder models can now significantly reduce the SiL
simulation run time during batch test by omitting code generation
prior to each SiL simulation. This feature is also applicable to test
sequences containing parameter variations.
If MTest detects that the configuration setting 'DefaultParameterBehavior'
of the test bed is set to 'Tunable parameters', the batch test
will generate the code only once for each test bed.
Hint: You can check this setting via:
get_param(gcs, 'DefaultParameterBehavior')
Please note:
The feature is only available in MATLAB R2015b and newer and Embedded Coder.- Exclusion of subsystem(s) from structural coverage computation
For a selected test object the user can now define an ignore-list
of subsystems. Those subsystems will then be excluded from the
computation of the structural coverage. The ignore-list will affect
both model coverage and code coverage.
Regarding code coverage the feature is available for
the TargetLink Code Coverage tool (TL only), as well as for
Testwell CTC++ (TargetLink and Embedded Coder).
In particular, this feature is beneficial in terms of testing
activities on integration level or subsystems utilizing library
instances.- Enhanced interface analysis
The interface analysis now supports two-dimensional array signals.
With this version Nx1 and 1xN arrays are supported.
In addition, MTest is now able to analyze array signals whose data
types are defined via custom data types defined in the Simulink Data
Dictionary.
Furthermore, the interface analysis is now also capable of handling
signals that are defined as bus elements within a Simulink bus object
that in turn is defined in a Simulink Data Dictionary as well.- Support of parameters defined in the Simulink Data Dictionary
Parameters defined in a Simulink Data Dictionary are now available for
use with MARS and MTCD. The parameters are read from the Simulink Data
Dictionary and their actual values are used for the general test group
initialization. Of course, they can be varied like any other parameter
from workspace.- New MTCD signal function 'gradientsteps'
With the help of the signal function 'gradientsteps' the user can
now gradually increase/decrease the signal's value similar to
the existing function 'gradient'. Now it is possible to define
for how long the intermediate signal values shall be held constant,
resulting in a step-like signal shape.- General update of MTest Specification Editor
When the MTest Specification Editor is opened the entire Eclipse project
will be refreshed, resulting in an update of the project structure and
the automatic detection of newly created files.- MTCD supports import of measurement data from ASCII and Excel files
In reference to the import of *.mat files, MTCD can now handle the
import of test data from ASCII files, as well as Microsoft Excel files.- New check for Test Project Protocol available
The introduced check compiles a list of test sequences that have been
evaluated manually. This helps to detect any differences between
automatic and manual evaluation and thus helps tracking manual results
(e.g. to avoid unintentionally overwriting test results).- The MTest Demo Project received an update
The major change is constituted by the addition of MARS and MTCD examples
in text format. That way, the user is now enabled to get a first
impression of the text-based MTCD editor and to get familiar with MARS
already at an early stage.- Acceptance of MES End-User License Agreement
Upon first start of the current MTest release, the user is asked to
accept the MES End-User License Agreement (EULA). At this, the user
has the option to view the EULA in English or Chinese language. MTest
will not start unless the EULA is accepted.- Enhanced data logging workflow
While working with logged signals defined in the testbed, the data logging
of Simulink and TargetLink signals can now be enabled/disabled separately.
For this purpose, two entries in the MTest main menu are available.
Here, available options do reflect the current state of the configuration.
Most important to note is that the Simulink data logging must be enabled
specifically by the user if a TargetLink user wants to log Simulink signals
(e.g. elements of Stateflow charts).
In the past, Simulink data logging was always enabled which could have
lead to problems in the case of signals being logged by both mechanisms.- Display of aggregated results in Manual Evaluation dialog
The aggregation of automatic evaluation results now includes the
results of the comparison to expected outputs defined in the MTCD test
description.- Selection of reference data
If the user selects any data for use as reference from sources for
which no data is available, the user receives more helpful feedback.- Handling of models with no subsystems
When selecting a model that does not contain any subsystem the user receives
feedback via a descriptive dialog box.
In addition the handling of the Data Logging Configuration for testbeds
that refer to models that do not contain any subsystems will now be handled
correctly.MTest v 6.1 (December 2018)
- Enhanced logging of internal signals (local signals)
Multiple enhancements for local/logged signals have been added
to improve the workflow, especially for test models where not
all necessary local signals are already switched on for logging.
The current configuration of signals to be logged can be saved
by choosing "Save from Test Bed" in the
"Execution > Data Logging Configuration" menu.
Using "Apply to Test Bed", the saved configuration can be
re-applied to the current test bed.
When a new test bed is created, the current configuration
is automatically applied to the new test bed. This allows the
test bed's logged signal configuration to be different
from the test model (and full batch mode application for the
configuration).
Logging of internal signals can be switched off by using
"Disable Data Logging". This switched off/on logging state is also
applied when creating a new test bed.- Calculating and reporting consumed tolerances in report and catalog
MTest now reports information about the consumed tolerances
for each signal comparison, in addition to defined tolerances in
the respective test sequence reports, as well as the signal
comparison catalog.
Now, you can see if tolerances were used for a Passed result and
what the maximal value deviation was during signal comparison.
Therefore, for a Failed result, the "necessary" value tolerance
for tolerated deviation can be seen directly.
This allows for a much quicker evaluation and review of signals with
used tolerances.- Defining equivalence classes for signals in MTCD
MTCD supports the definition of equivalence classes of signals
to enable convenient documentation of this information.
The parser and the validator support the new syntax. Templates
are available in the content assist. See the MTest User Guide for
examples.
(In later versions, the documented equivalence classes will be
used for extended features.)- Enhanced export of xml and new junit (xml) format
The xml export is now available. The xml file contains
test sequence information and test results.
The junit xml (separate format) contains information on test
sequences. It is used in the MTest step of the MES Jenkins
Plugin and can be used (only) with the MTest CI license.- Deleting project elements
Via a menu option, single or multiple test sequences, test groups,
test objects, or test models can be selectively deleted from an
MTest test project.- Enhancements of MTest Specification Editor
The work status of test cases and groups in MTCD is now supported
by an enumeration of valid entries. By entering the appropriate work
status you are able to trace your test case definition progress.
Previous work status entries in existing test specifications
- defined as strings - can simply be transformed by using the quick fix
in the error message.
The workspace folder was renamed from "marsWorkspace" to
"editorWorkspace". Existing test projects are adapted automatically.
Signatures for MTCD functions are now provided in content assist.
Negative exponents are supported.- Added new checks for Test Project Protocol (TPP)
Test sequences with evaluated assessments: Finds assessments
that are active but have no evaluation data.
Requirements are imported to test object: Finds test objects
with no requirement document imported.- Setting testability review status during requirements import
When the testability property can be automatically calculated during
the requirements import (by setting the right options and availability
of data), the testability review status is set to 'reviewed
(derived)'. This 'reviewed (derived)' value is interpreted as reviewed
in the requirements metrics (in requirements catalog and data export).
In this way, the manual review of an automatically calculated
testability status is no longer necessary.- Simulation execution in base workspace supported
MTest normally performs simulation in a secure function workspace,
preventing interference with values existing in the base workspace by
chance.
Now, you can select the option simworkspace = 0 in MTest_UserPrefGUI
to perform test sequence simulation in the base workspace.- Test objects with function call outputs directly supported
Subsystems with function call outputs can now be used directly
as test objects. During test bed creation, additional blocks are
included around the test object to convert the function calls to
scalar signals.- (Alpha) MTCD test data viewer with quick refresh of inputs and expected outputs
When defining signals in MTCD, you can view your input and expected
output signals quickly by Sequences/Test Data Viewer. After changes
have been made in MTCD (and saved), the signal figure can be quickly
refreshed by figure menu MTest/Re-Import Test Data and Refresh Plots.
The same signals are shown in the figure (using the newly imported
data from MTCD). The list of signals to view will be remembered for
the test object and can be re-used for viewing signals of other
sequences (of the same test object).- MARS value tolerance for comparisons
Value tolerances can be applied for comparisons in formal requirements
written in MARS. The generated assessment then tolerates
the desired deviation for this comparison. Value tolerances can be
defined as absolute values using "[+- <value>]" or relative values in
in percent using "[+- <value>%]" after the right-hand side expression
in a comparison and for the expected system response. For <value>, all
arithmetic expressions known to MARS are allowed, including signals
changing over time.- MARS vector parameter access
For parameters that are vectors, it is now possible to access the
indices using parentheses. E.g.: "parameter PARAM(1)" to access the
value at the first index of the parameter PARAM- MARS enhancement for combined state and event Trigger
In some cases where "or" was used in the state trigger, the combined
trigger was incorrectly calculated, which led to an unexpected
behavior.- Parser enhancements for broader use of expressions
In earlier versions, it was, at multiple locations in MARS, only
possible to define single values and not to use signals or parameters
or arithmetic operations. With multiple parser enhancements, it is now
possible to use all arithmetic MARS expressions in most contexts,
such as timing definitions.
(E.g.: "... for 3 + parameter delay seconds" or
"...SHALL be true within 1 / parameter controlFreqInHertz seconds")- Workflow enhancements
- Better recognition of local Excel versions for used separator
and language, preventing some "unusable" cell selection in Excel
files (especially the work status column in MTCD).
- In MATLAB R2015B, there is a bug preventing the full use of the model
coverage functionality. MTest applies the necessary work-around
(thanks to TMW) for the appropriate MATLAB versions, enabling model
coverage and repeated access to the model coverage options dialog.
- When adding a new test evaluation configuration, the example
configuration in MTest_SystemOption in the demo directory is used
again (and not an empty configuration). In this way, the default test
evaluation configuration can be pre-configured by adapting
MTest_SystemOptions in the demo directory of your MTest installation.
- The documentation of varied parameters per test sequence in the
test report now works again (it was "refactored" in 5.2/6.0).
-Analysis of logged/local signals in 6.0 could lead to a loop
when multiple referenced models were used in the test object and
one of the referenced models was referenced more than once.
- In Batch mode deleted test sequences no longer stop the execution
of batch actions of subsequent test sequences.MTest v 6.0 (September 2018)
- MTCD text editor as part of MTest Specification Editor (Eclipse)
MTCD can be written and edited in a dedicated Eclipse/text
editor - the MTest Specification Editor (also used for MARS).
This MTCD editor has no limits for the length/size of a test
sequence or parameter definitions (as Excel has).
The editor provides templates for test group, test sequence,
signal names, parameter names, MTCD functions, and more.
The written MTCD is validated online and helpful hints are shown.
An outline view and navigation and the many other usual editor
features are also available.
The Excel MTCD can be converted to a text MTCD (menu).
The MTCD specification by Excel is still available.- Automatic adaptation to changes to the test object interface
When changes are made to the test object interface, MTest reports
these changes (diff analysis: new/renamed/deleted signal).
These changes can be checked, mapped (i.e. from new to renamed),
and confirmed. Using this mapping function, the interface changes can
then be disseminated to MTCD, MARS, reference data, test setup,
tolerance definitions, and further definitions in MTest.
In addition to storing the current interface definition, MTest also
stores the history of the interface (different baselines). This way,
interface changes can be mapped on the basis of one of the
previous interface baselines.
A mapping GUI provides an intuitive way of mapping and selecting
dissemination actions.- Batch functions return extended status information
The MTest batch functions (MTest_BatchProject and MTest_Batch on
a lower level) return extended status information. This
aggregated value for all performed batch actions provides one
result regarding everything OK (1), something not OK (-1), or
nothing done (0).
For CI (Continuous Integration), a success criteria can be used
to provide a success value even when the results of a test
evaluation were 'failed' (but nothing aborted or errored). This is
useful to get a stable result in Jenkins, even when not all
evaluations have the result 'passed'.
A CI license for MTest is necessary to use this feature.- Batch configurations/selections in MTest_BatchProject
Batch selections saved in Batch GUI (in a mat file in test project)
can be used to configure a batch run in MTest_BatchProject (and in
the MES Jenkins Plugin).
These batch selections must be created/saved in MTest 6 or newer.
Batch selections should only be used for specific use cases.
The direct definition of test models/objects and batch actions for
MTest_BatchProject is much more visible.- MTest step in MES Jenkins Plugin
The MES Jenkins Plugin contains an MTest step. Please see the
documentation of the MES Jenkins Plugin.
A CI license for MTest is necessary to use this feature.- MARS enhancements
Defines representing a scalar value not dependent on simulation
data are now properly set as parameters and are used as such in
the generated assessments.
The number of supported arithmetic functions has greatly improved.
Additionally, it is now possible to use arbitrary function names for
user-defined functions.
The "^" operator is now available to potentiate in arithmetic
expressions.- Inclusion of test sequence report graphics directly in HTML report
The graphics of the HTML test sequence report can be included inside
the HTML report (as base64 data).
In this way, the number of files in the test project is now much lower.
This graphic inclusion can be switched off in MTest_UserPrefGUI by
PropGUIDefault.report.embedimagesbase64.- Option to skip extra interface analysis if model did not change
Before an interface analysis is performed, MTest can checks
if the model to test really changed since the last interface
analysis. If no change was detected, the still current interface
information is used.
Multiple conditions are checked (if any changed, interface
analysis is performed): modification of model, modification of
referenced models, modification of Testsetup files.
This option must currently be enabled in MTest_UserPrefGUI by
execution.forceinterfaceanalysis = 0 (disable force interface analysis)- User Guide v 6.0
The MTest User Guide contains descriptions of the interface changes
workflow.
The MTCD text editor in Eclipse is explained as part of the MTCD
Chapter.
Details about the extended batch features are described in the Batch
Chapter.- Workflow enhancements
A few of the settings in MTest_UserPrefGUI were changed to reflect
the settings most customers of MTest use (this is a major release).
Please check your previous usage of these settings and adapt if
necessary.
A paused batch run can be resumed again (introduced in MTest 5.2).
The signal catalog shows all test groups (and not just the last one).
The baseline (version info) of the requirements document (RSD) is
shown in the start of the requirements catalog (it was already
visible in each requirement).
During test bed generation and the TL module enhancement, further
output parameters are transferred to the test bed: InitialOutput
and OutputWhenDisabled (reset method).
The MQC XML export was extended to support different naming of
model coverage methods in different MATLAB releases and really
exports the structural coverage values of multiple test objects.MTest v 5.2 (June 2018)
- Extended tolerance definition for all signal comparisons
It is now possible to define signal comparison tolerances
separately for each signal and each evaluation method
(for regression, back-2-back, expected output).
At the same time, the default tolerance value for each signal,
which is used for all signal comparisons, is still available.
Additionally, it is possible to define a global
(non-signal specific) default tolerance for all the available
signals.
The evaluation-method-specific tolerance has the highest
priority. The signal-specific default tolerances have a
lower priority. The global default tolerance for all
signals has the lowest priority.- Relative tolerances for all signal comparisons
When configuring the tolerances, you can now also define
relative tolerances. If both absolute and relative
tolerances are defined, MTest will use the larger tolerance
for each time step of the sequence.- Batch mode with Pause/Proceed and Stop
The Batch GUI displays a Pause and a Stop button after starting
the batch run. This makes it possible to pause or stop the batch
run.
If Pause is pressed, the batch run pauses after the current
batch action has been completed.
If Stop is pressed, all further batch actions are skipped after
the current action. This is also shown in the MATLAB
command window and the Batch report.- Batch mode is now silent
In batch mode, the model coverage report no longer pops up.
A newly generated test bed is generated in the background
during batch test. The catalog generation in batch mode does not
open any wait bars.
In this way, the batch mode can now run in the background as
most actions will no longer open a pop-up window that changes window
focus.- Batch GUI remembers settings after closing
The Batch GUI saves the current settings (selected sequences
and actions) when closed. Upon reopening, these settings are reloaded.
This allows you to pick up where you left off last time.
If you want to reset all the settings (selected actions and test
sequences), there is a button in the Batch GUI that allows you to do
this.- Batch GUI refresh
If the project structure is changed, the batch window will
automatically update as soon as you click on the window. This
also works if you change the test project outside the batch window.- New MTCD function sinesweep
The sinesweep function implements a sine with variable period
and variable amplitude.- Result distribution in requirements catalog
We added the result distribution inside the requirement catalog
showing the status of the requirements in more detail. It behaves
in the same fashion as in test sequences and assessment catalogs.- Workflow enhancements
We improved documentation of errors in the test setup file so that
a resulting incorrect test execution can be readily seen. This
helps with error traceability.
We also made sure that the MiL simulation of a TargetLink model is
possible without having to have a TargetLink license.MTest v 5.1 (March 2018)
- MARS: MTest Assessable Requirements Syntax v 1.5
The MARS language has been slightly refined, to better reflect
natural language. A few syntax elements were marked as deprecated.
Please open your MARS file in the MARS editor and check for
deprecated elements (look out for yellow warning markers). These
deprecated elements are still supported, but will be removed
in the future.
Assessment generation has been improved with respect to multiple
adjacent trigger events. We increased robustness of assessment
generation (e.g. more secure handling of various test data dimensions,
representation of comments). Also, the generated assessments adhere to
the standard assessment template.
The MARS Quick Reference Guide (formerly known as cheat sheet)
has been improved for better readability and comprehensibility.
The MARS editor now starts more quickly and is more streamlined to the
MARS workflow.
The content assist/templates of the MARS editor have been extended and
improved.- Introduced new MTCD Parser, based on Xtext technology
During the MTCD import, you get much better feedback on incorrect
commands and specification (exact line and cause of incorrectness).
The new parser is stricter by design. If you are using "custom" things
in MTCD (which produce messages in the new parser), please tell us.
We will extend the syntax, if useful.- Test Project Protocol extended with new checks
The Test Project Protocol has been extended by a large number of new
checks.
Give it a try when you think your test project is done. It might
point to some things that are not really finished yet.- Enabled evaluation of multiple regression tests
A recent bug prevented the execution of multiple regression test
evaluations in the same configuration. Now, the simultaneous comparison
of MiL outputs and SiL outputs to reference data can be performed.- TL code coverage aggregation also for separate TL Function subsystems
The TL code coverage aggregation was extended to handle TL Function
subsystems appropriately.- Custom storage location of test models
The location of your tested models is no longer limited to one
specific directory. You can select models to test from any directory
when adding a new test object.
Additionally, the location of the model to test can be reassigned when
moved to another directory.
Models must still have a unique name within an MTest project.- Extended import of MTCD/Testona in Batch Mode
The MTest Batch import can now perform the initial import of
test sequences. When selecting at least the complete test
group (or test object/model) in the Batch GUI, all test sequences
in this MTCD (or Testona/CTE) file are imported (even when new).
Afterwards all further batch actions (sim, eval, ...) are performed on
all these test sequences.
This initial import works with MTest_BatchProject too.
For these imports please define the default simulation step size
(used for discretization of imported signals too) in the global
parameters setting in the project directory (see file
mtc_DefineMTcOptions_SimParaGlobal in MTest/demo directory).- Workflow enhancements
When a model is already open before MTest is launched, this model
stays open after changing models in MTest or closing MTest.
Opening of TL models with DDs that include other DDs by relative
path is now handled more robustly (MTest really opens these models
from the model directory). Additionally, the DD is opened afterwards
explicitly.
In the generated test bed, a vector of function call signals does
not get a Data Type Conversion block afterwards (which all the other
vectors on the input side get and need).
During simulation and during TL code generation the test bed is
just loaded (i.e will not be opened). Thus, no test bed will pop up.
This enables batch testing with the focus being kept. However, if
you regenerate the test bed, and/or you perform model coverage
the test bed, and/or the coverage report popup, respectively.- User Guide v 5.1
The user guide now also includes the extended import of sequences in
Batch and some information regarding the new MTCD parser.
The MARS Chapter contains the new Quick Reference Guide and
extended description of multiple trigger behavior.MTest v 5.0 (December 2017)
- Expected outputs as additional evaluation method
You can now use the MTCD test description file to define
expected output values. Once the method is activated in
Configure Evaluation, MTest automatically compares the
output data to the values of the defined expected output data.
The method allows the specification of expected values
for either the entire test sequence or particular intervals
only.- Extended model coverage support for referenced models
Model coverage for models with one or multiple referenced
models is now supported. The user can now specify which
referenced model(s) shall be considered for obtaining the model
coverage. The reports and catalogs detail the summed up coverage
for all referenced models or for the selected models
(see model coverage settings).- Reporting of applied tolerances
In case of test evaluation by signal comparison (regression,
back-to-back, expected outputs), the applied tolerances are
presented in test sequence reports and signal comparison catalog.- Full vector/matrix signal support in interface analysis for TargetLink buses
The interface analysis of TargetLink busses has been expanded to
handle used vector and matrix signals inside the TargetLink bus.
Now, the signal hierarchy and the compiled data types are combined
for a full TargetLink bus interface (and not as effective interface
as other inputs).- Improved TL Module enhancement
The automatic generation of a TL subsystem from a subsystem was
improved. The analysis of the source of a signal handles more
cases in a more robust way (for instance Mux blocks and their
respective source).
If you had test objects where the automatic enhancement did not
succeed please rebuild the test bed and check again. Now the
automatic TL module enhancement handles more cases than before.- No need to restart MATLAB if license fails once
If the checkout of a license fails once, a restart of
MATLAB is no longer necessary. Simply configure the license
correctly and restart MTest.- Enabled model coverage support for R2017B
MTest now supports the toolbox Simulink Coverage(TM) (as introduced
with R2017B) for obtaining model coverage.
MTest removed new compatibility warnings regarding R2017B too.- Enhanced conversion from Signal Builder to MTCD
When converting from signal builder blocks to MTCD, some rare
cases are handled more robustly. This includes the start of ramps, the
end of a ramp starting at t=0, the handling of very short
time steps, and the assignment between changes in the signal builder
and the corresponding test steps in MTCD.- MTCD test sequence import ordered by ID
The import order of MTCD can be switched from the default order of
the MTCD file to using the IDs of the test sequences in the MTCD file.
In this way, the test sequence data is reimported into the same
sequences, even if sequences were added/removed in the MTCD file
(especially important when using reference data for regression test).
See the User Guide for a description of the ordering logic.
This feature can be switched on in MTest_UserPrefGUI by
PropGUIDefault.import.sortsequences = 1.- View local data in View Test Data and Outputs
Beside input/test and output signals now local signals are available
in Documentation/View Test Data and Outputs too.
The signals are grouped by type (first input, then output, then local
signals). Each group is sorted alphabetically. In this way signals can
be found quicker.- (Beta) MARS MTest Assessable Requirements Syntax v. 0.5 Support for Lookup Tables The MARS syntax now allows expressions referencing a lookup table (1D and 2D). The generated assessment uses the assessment functions MTa_mapLine and MTa_mapField.
Assessment Generator Robustified
State driven requirements with state triggers that are not
dependent on a signal of the test object are now either triggered
or not triggered for the complete test sequence, depending on the
specified trigger condition.- Workflow enhancements
The associated requirements document (RSD) can now be called from a
drive other than the test project's drive. MTest then use an absolute
path.
Higher robustness for Simulink objects in parameter
definition (value vs. initvalue).
When analyzing parameters for a test object, the value
of the parameter is no longer overwritten by an empty value
in the TargetLink DD (or DFF), if found in both places.
Logging internal TargetLink signals in R2015B and newer in MiL
now works using the TargetLink logging mechanism. For these releases,
the simulation of test sequences in MiL is performed in base
workspace. Only in this case does the TargetLink MiL logging work
(known feature).
Assessment framework utility functions can now handle non-double
signals and parameters.
Signal history plots in test sequence reports are placed in horizontal
alignment.
When a test object has no inputs (just parameters) the generation of
a new MTCD file works now.
During test bed cration for TL test objects relative search paths for
'IncludeFileSearchPath' and 'SourceFileSearchPath' in TL DD are
adapted, if an relative path is used (since MTest 3.0). Because of
problems in TL 4.2 the trailing ; was removed. BTW, the relative path
must start with .. to be recognized as relative path.- Update of user guide The user guide was extended with a chapter on MARS including introductory examples and with a chapter on expected outputs. The reporting of applied tolerances is included in the respective sections. Our assessment driven test design is documented in a section as well.
MTest v 4.4 (September 2017)
- Overview of model and code coverage metrics for complete project
At the beginning of the test sequence catalog, MTest displays
all model and code coverage metrics of all test objects in
one table. This provides you with a comprehensive overview
of all structural coverage values for each test object. You
can even see which test objects were not simulated using
model or code coverage.
The heat map coloring provides additional feedback on the
state of the structural coverage.
See the User Guide for a quick introduction to all the
information displayed (if necessary).- New Test Project Protocol
Large test project, long catalogs, many passed, some failed.
What is missing in my test project, what still needs to be
done, what should be reviewed?
The new Test Project Protocol provides answers to these and
more questions. Think of it as a "to do and open points" list
(at the moment).
We are interested in your feedback (and we have a lot more
ideas for the Test Project Protocol).- Function caller block handling in interface and test bed
The function caller block and the associated functionality
is included in the test bed, even when parts of it are
defined outside of the test object.
The function caller definition inputs are included
in the test object interface as additional inputs.
Remark:
The function caller block is different from a function call.- More robust support of vector inputs in test bed
In the test bed vector input, signals receive an additional
signal conversion block after the MUX block. Otherwise,
Simulink throws messages before test bed simulation.- (Beta) MARS MTest Assessable Requirements Syntax
The User Guide contains a first chapter about the use of MARS. It
explains the structure of the formal requirements and provides
a few examples so you can get started quickly with MARS.
The installation instructions describe the MARS system
requirements.- Extended documentation of MTa_checkReaction in User Guide
MTa_checkReaction is the most used assessment function. In
the User Guide, the description and the examples have been
extended and reworked for even better understandability.- More precise output of parameter values
The values of parameters are documented more precisely (more
significant numbers) in the test sequence report, parameter
report, and assessment analyzer.
Up to 16 digits are documented (if necessary).- Extended Embedded Coder SiL support
Problems during EC code generation produce a NOT OK during
batch run (silsim with code generation) and during interactive
code generation (Execution Options/SiL/Apply&Build).- Added MTCD function sawtooth()
MTCD now provides an action function for generating:
- sawtooth/saw wave signals
Use this function for counters, keep alive signals, and similar.
See MTest User Guide for further information.- More precise setting of ramp end values in MTCD
The end value of each ramp or sinramp in MTCD is set exactly
to the given value.
Previously, due to floating point calculation of the final
value, its exact value could be missed by a very small magnitude.- (Alpha) MTCD XText Parser for validation
If you want to use it, please download the extended release file:
http://mtest-classic.de/fmtestrf8relv3xnpw/...
MTest_4_4_Release_withMTCP.zip The MTCD XText parser is now directly available from the MTest
menu bar (Sequences/Validate Test Description) for validating
the current MTCD file.
This validation is an optional step (in this release). Please
use it when the import of MTCD is not successful. The XText
parser provides detailed and precise messages in areas where
the MTCD syntax in your file is not correct.
The new parser is stricter by design. If you are using
"special" things in MTCD (which produce messages in the new
parser), please tell us. We will extend the syntax, if useful.
The MTCD import is still done with the current/previous parser.- Workflow enhancements
For enabled/triggered test objects with bus inputs, the
test bed generation now works. Previously, the connection
of the bus creation block and the test object did not work.
(Enable/triggered subsystems with scalar vector inputs
already work and (nottriggered) test objects with bus
inports also worked.)
The navigation menu in catalogs now shows all the different
catalogs all the time (previously, the current catalog was
not shown in menu by design).MTest v 4.3 (June 2017)
- More convenient access to parameters in assessments and MTCD
The access to parameter values was generalized in MTest.
Now even parameter values of class Simulink.Parameter or
other Simulink.Objects are directly accessible via
the name of the parameter (or parameter structure).
Please check your assessment scripts. If you
accessed Simulink.Parameters via PARAMETERNAME.value = 5,
then you must adjust these calls to the direct method:
PARAMETERNAME = 5.
In MTCD the default values are used in template even for
structures of parameters and objects in parameters.- Parameter diff in sequence report for parameter objects
The parameter difference reporting in the sequence report
(showing used parameter variation) shows also differences
for objects in parameters (especially Simulink objects).- Interactive support for parameter setup for new test objects
When a new test object is added you are asked if you
want to set up parameter loading (TestSetup) for the
test model (providing default parameter settings for all
test objects selected out of this test model).
This prevents repeated definition of the same TestSetup.m
for each test object. This tool driven configuration
removes one often seen problem in setting up new test
models and test objects.- Extended generation of Embedded Coder SiL test beds
The automatic generation of Embedded Coder SiL test
beds was reworked. Besides an S-function the creation
of a SiL-block test bed is done for ert-targets.
See the User Guide for more information and the different
possibilities.
CTC++ code coverage for EC test beds is enabled as
well and a few interesting effects could be resolved.- Identification of out-dated manual evaluation results
Manual evaluation results older than an automatic
evaluation result (regression, back-to-back, assessment)
are marked in the test sequence catalog and the test
sequence report by a recognizable background color.
This enables a comfortable search for these manual
results if necessary.- Test sequence report shows name of test group
At the top of the test sequence report the test group
name is documented as well.- Extended interface analysis and test bed creation
Triggered, enabled, function-call (trigger with function-call,
sample time type could be triggered or periodic) and
if-action/case-action subsystems were supported
for SL and EC already.
The automatic enhancement of a subsystem out of a TL
subsystem worked for triggered, enabled, and function-call
subsystems already.
We now support test bed generation including TL enhancement
for subsystems containing If-action subsystems (transfered
into enabled subsystem), thus keeping the enable property. During TL subsystem enhancement TL functions were only enhanced
(MTest assumed, that all in/outports were TL-In/Outports
already). Now MTest checks first, if there are none TL ports.
Then the module gets a full wrapper with added TL in/outports. The interface analysis of nested busses defined by bus objects
was enhanced. In sub busses the real bus name is used (previously
we used the busdatatype name, which works often, but not always).
In these cases the bus name is also used during test bed creation
to name the sub bus in the bus building block providing extended
bus naming in more cases.- (Alpha) MARS MTest Assessable Requirements Syntax
A formal implementation method for requirements is available.
It is based on a powerful syntax close to natural language.
The syntax is based on the EARS pattern and introduces
further additions to describe state-based and event-based
requirements.
The really interesting part is, that MARS automatically
generates executable assessments out of the requirements.
This first release of MARS is to get a grip on the
very interesting possibilities. We use it already to
generate nearly all our new assessments.
If you want to try it out, please contact MTest support
to get an introduction and to show you some examples.- Workflow enhancements
A possible *.req file besides the test bed is removed
before generating SiL EC code preventing an
interactive dialog in R2016b and newer.
The parameter report (documentation of parameters
for test object) is regenerated, when a test
sequence report is generated via the menu command or
using the main GUI.
Test objects without inputs (i.e. loading of parameters)
and/or without output signals (calculating internal
signals) can now be handled in MTest.
Handling of the new model coverage configuration in
R2016a and newer included.- Maintenance release 4.21 (May 2017)
Model coverage was sometimes not collected with
the recently introduced Coverage Framework.
More robust handling of interface analysis and
test bed creation for test objects with boolean
inputs and for output buses without bus creator
in front of output.
User Guide update regarding update to MTest 4.0
and MTest 4.2 (rerun projects).
More robust handling of evaluation configuration
in GUI when defining separate settings than for
the whole test object.
More robust enabling of manual evaluation menu
command.
Check and handle maximal lines of initialization
of MTCD file necessary for test objects with
large interfaces (there is a 255 lines per cell
limit of Excel).
Subsystems in test model with new line in name
are handled by Add New TestObject list dialog.
Test objects with new line in name are pointed
out by a message.
Assessment Analyzer handles NaN more robust.
Test Catalog creates more robust HTML removing
unnecessary white space.MTest v 4.2 (March 2017)
- New structural coverage framework (model/code coverage)
The result metrics of structural coverage (model and
code coverage) are shown in detail in the respective
test sequence report and the test sequence catalog.
The used coverage values for the test sequence are
shown in test sequence report.
Aggregated coverage values for all test sequences of a
test object are shown in the test sequence catalog.
To use the new structural coverage with MTest, please
rerun existing projects anew (simulation including
model/code coverage enabled).- View Model opens test object in model
The View Model menu command and the View button in
the Testobject panel each open the model file and
change the view to the selected test object.
In this way, the selected test object can be quickly
located even in large models.- Requirements shown in new requirements catalog
MTest provides a new requirements catalog
(similar in structure to test sequence and
assessments catalog).
The requirements catalog lists all requirements
related information including links to related test
sequences and assessments and requirements related
metrics.
The requirements catalog replaces the requirements
parts of the RCT report.- Extension of catalogs by requirements information
The sequence and assessment catalogs now contain
(more) requirement related information, including
full linkage to the respective parts of the other
catalogs or sequence report.
This replaces the test sequence and asessments
parts of the RCT report.- Extension of catalogs by metric information
All catalogs contain the respective metrics
information (some of them formerly found in the
RCT report).
At the beginning of each section the dedicated
metrics are shown in tabular format (number of
requirements/assessments, covered reqs/assessments,
work status and so on).- Refactoring of RCT framework
All the functionality of the RCT framework was
refactored. Data is saved in another way, some
extended functions were prepared. The previous
functionality is still available. Old data sets
are automatically converted to the new format.
If you used any of the MTr* functions (especially
MTr_RCT) directly, please contact us for adaptation.
To sum it up: it is much quicker now, there is
no extra RCT report ("just" a new requirements
catalog) and metrics are displayed where expected.- GUI for Evaluation configuration
All the options for evaluation (regression and
back-2-back test and assessments) can be configured
in a GUI supporting all the different combinations
of the options (Configure in Evaluation panel).
The GUI uses the same configuration file format and
place as before (loading and saving).- GUI for manual evaluation
In addition to the automatic evaluation methods
a manual evaluation of test sequences is possible.
The manual evaluation has a higher priority than the
automatic evaluation methods (assessments, regression,
back-2-back).
Manual evaluation can be used as exclusive evaluation
method or to override automatic evaluation results.
The GUI provides quick review and setting of manual
evaluation for all test sequences of the current test
object.
The main GUI shows an additional aggregated evaluation
result, which includes manual evaluation.
Reports and catalogs show the manual evaluation too.- Requirements compliance metric uses all automatic evaluations
The requirements compliance is the aggregation of
assessments, back2back and regression using traceability
information. (Previously only assessments were used.)- Display of Simulink objects in parameter report
The model parameters can contain different types of
Simulink objects (Parameter, Signal, BusObject,
AliasType, CustomStorageClass, Enumeration, ...).
These variables are fully documented in the parameter
report of the test object and in the changed parameter
section of the test sequence report.
For enums a compact one line documentation is applied.
For the other objects all fields of the object are
documented in a structure like way.
If you want to use this documentation facility,
look at prprintf.- Conversion of CTE test specification to MTCD
CTE test specification can be converted directly to
MTCD test specification. See MTest_cteConvert2mtcd
and accompanying files in bin/AddOn/CTE2MTCD.
The relevant information in the cte file is transfered
to the respective MTCD syntax and placed in an Excel
file.
Afterwards you may use the MTCD to add further info
(ID, covered requirements, work status, ...).- Workflow enhancements
Message, if no assessments found after pressing
Open button in Assessments panel.
Additional option in MTest_UserPrefGUI to enable
copying of TL AddFile blocks during test bed
generation (previously this had to be done in
MTest_PortsExtraHandling).
The aggregation of CTC++ code coverage data from very
large sets of test sequences was changed from command
line call to parameter file call. In this way the
number of sequences to aggregate CTC++ coverage data
for is no longer limited by the length of the internal
command line.
The robustness of the batch test was enhanced. An
error during test bed creation or interface analysis
does not stop the execution of following batch
actions or test sequences.
During test bed creation the name of the output signal
of the output data type conversion (DTC) was prefixed
with mo to prevent hidden data type conversion messages
from a possible matching signal object.MTest v 4.1 (December 2016)
- Inclusion of test sequence status during import from MTCD
During import of test sequences from MTCD the test sequence
work status is now included. Previously the work status
was only available in RCT.
The work status is shown in test sequence report (if
applicable).- Logging of SL internal/local signals
(Beta) Logging of local SL signals in MTest (without
any extra blocks or configuration). Just enable
logging of signals in the signal dialog, everything
else is handled by MTest. These local signals are
included in the test sequence report and can be used
in assessments.- Transparent handling of filter for TL DD syncronisation
TL requests that parameter values are identical in DD
and workspace before code generation and simulation.
This synchronization is handled by MTest using TL sync
function (see MTest_PortsExtraHandling).
Sometimes this syncronization must be bounded to
defined parts of the DD tree.
If an m-file MTest_synchronization_filter.m is in the
MATLAB path, it is used as a filter.
If the file can not be found by MATLAB, the standard
synchronization will be performed.- Close all models opened by MTest after use All models which are opened by MTest (models to test and test beds) are closed during change of selected model and when closing MTest.
- Enabling of TL module test for modules with TL BusIn/Outports The automatic module test (creation of test bed) works for subsystems with TL In/Outports and SL In/Outports. Now it works for subsystems with TL BusIn/Outports as well. In this case, no effective interface analysis is performed (MTest just takes the full interface of the subsystem). Note: Subsystems with SL (Bus)In/Outports still can not be enhanced to a TL subsystem.
- Further enhancements to administrative data management For the administrative data handling functions further internal enhancements were implemented. Some calls to the new functions were adjusted to handle code refactoring (get global simulation options, create catalog without eval data, reports without eval data, selection GUI of models for Add TObj, ...).
- Compatibility enhancements
Due to changes in different MATLAB releases some
refactoring was necessary:
- multiple calls to Batch GUI in R2015B and newer (no
persistent handles in MTest_BatchConfig)
- location of plot legend in R2016B (no integer in
legend location)
- Workflow enhancements
- During MTCD/CTE/TESTONA import all previous test data
are cleared first. Thus, no artefacts of previous
imports remain in test data. On the other side, no
extension of previous import data is possible (which was
a feature quite some years ago, but is no longer used).
- Removed menu bar entry: Aggregate model/code coverage - These menu options became obsolete, since they only applied to SL and TL models. Moreover, coverage aggregation is handled by the the batch mode entirely.
- Update of user guide - The user guide received a major update reflecting the new work flow and features of MTest as they have been introduced in the releases v 4.0 and v 4.1
MTest v 4.0 (October 2016)
- Complete overhaul of main GUI
The main GUI was reworked to represent the complete testing
workflow. The main tasks can be started by one click.
Important evaluation and status information and not yet
performed tasks can be seen directly.- New evaluation framework introduced
The new framework enables a better data handling of
regression, back-2-back, and assessment evaluation.
This enables a more detailed (test sequence report) or
a more aggregated presentation (catalogs) of evaluation
information.
The new signal catalog is another new feature out of
the evaluation framework. More to come in future releases. Attention: All evaluations must be re-performed by
MTest 4.x to make full use of the introduced feature.- New signal catalog for aggregated regression/back-2-back results
The all new signal catalog provides compact information on
failed signal comparisons, which better supports regression
and back-to-back evaluation.
For each signal the failures over all sequences of one test
object are contained in one place.
Please try it out, we really love this one (further features
will be added in the next releases).- Extensive overhaul of handling of administrative data
A lot of quite old internal methods were completely rewritten
and extended. Now many internal steps are much easier and more
clear at the same time.
An added benefit is the again much smaller handling time.
Try for instance catalog generation on your own.
If you ever used any internal calls into MTest m-files: they
probably do not work any more. Please contact MTest support.- Selection of specific test object(s) in MTest_BatchProject
The execution can be narrowed to specific testmodels and to
specific testobjects (per testmodel).
See help MTest_BatchProject for examples.- Change of file extension for TL data files
We changed the simulation mode specific extension of data files
for TL files
MiL: fl_tl -> mil_tl
SiL: fx_tl -> sil_tl
PiL: target -> pil_tl
See below for steps to adapt your projects for this change.- Major release:
This is a major release. We changed a few things by design.
The step from an older MTest version is quite painless.
- Rerun all simulations (inkl. model/code coverage)
- Rerun all evaluations
- Regenerate reports and catalogs
If you used reference data, convert the naming of the ref data
files by using MTest_ConvertRefFiles.
The way back from 4.x to an earlier version is not granted
but feasible.- Update of integrated FlexNet Publisher libraries
The integrated FlexNet Publisher libraries have been updated
to version 11.14.0.0 (same as MXAM Drive 4 and M-XRAY 3.1).
This update enables the license module to read dongle ids on
systems running the WibuKey driver in version 6.32 or above.
Older WibuKey driver versions are still supported.
If you experience problems with your Flex License Server,
please revert to the previous version of the DLLs in a previous
MTest version (in MTest/lismo) or contact MTest support.- Borrow a floating license
Borrowing a floating license is possible. Please look at the
information in MTest/readme_installation.txt.- MTest Tool Classification Kit available
Please contact sales@model-engineers.comMTest v 3.52 (August 2016)
- Interface analysis extended
Busses with only one signal are recognized as bus now.
Rate transitions and signal conversions are handled
during analysis of effective interface.
BusCreators in front of a (bus) output of the test object
are handled.- Test bed generation
The layout of the bus creation subsystems was enhanced.
Now the layout can be recognized even for multiple
multi-level busses.- TL Plot Overview windows not opened in batch mode simulation
If signals are logged in TL models, then TL opens a
TargetLink Plot Overview window during simulation.
In MTest batch mode this plot overview window is not
opened during simulation (logdata.plotchannels are set
to [] instead of -1).
During test bed generation an TL plot window can still
be opened- CTC++ config checker enabled
The CTC++ configuration is checked before the first call
to CTC++. If anything is not correct, a detailed message
is provided and CTC++ is disabled.- MiL simulation with TrueDoubles in Batch enabled
Added an option in MiL simulation to change all numeric
data types to TrueDoubles (aka boolean stay boolean).
Currently this option is available in batch mode only.- Extended handling of test objects with loads of signals
To handle more than 800 signals as input interface the
test bed is generated in a more compact manner.
The MTCD template generation to Excel accounts for more
than 200 signals and parameters (to handle the Excel
cell size limits).- Enabled display of Simulink.Parameter in Assessment Analyzer
The Assessment Analyzer can now display Simulink.Parameter.- Fixed View Test Data no input signals shown
In 3.5 (only) the inputs were not shown in View Test Data
and Outputs.MTest v 3.5 (June 2016)
- Assessment activation on test sequences by linked requirements
The activation (or non-activation) of assessments
for test sequences can be easily limited according
to linked requirements. Assessments are only active
if the test sequence is linked to the same requirement
as the assessment. One matching requirement is sufficient
for activation.
This assessment scope can be configured for all assessments
in Evaluation/Configure Test Evaluation by the additional
option AssessmentScope. In each assessment the general
setting can be changed individually by static field Scope.
Additionally, you may also limit the scope of an assessment
to specified test sequences. See MTest User Guide for more
details.
The generation of new assessment templates is extended
to generate the new fields into the assessment template.- Simulink.Bus objects recognized during interface analysis
If the data type of a port/signal is a Simulink.Bus
object, MTest uses the information of the object for
interface analysis (multiple bus levels, signals, and
data types). It is not necessary to compile the model.- Interface analysis without compiling model
MTest now analyzes the test object interface even
when compiling (updating) the test model does
not succeed. This is especially important for
unfinished models where only some subsystems will
be tested (aka interactive test bed testing).
Buses will only be recognized if a Simulink.Bus
object is available (see above).
Data type of signals will be standard data type and
not the CompiledDataType.
A message is provided in the command window to notify the
user of the uncompiled analysis mode. Some signal attributes
only available in compile mode are missing in the interface
description (especially CompiledDataType).- Check of CTC++ installation
When the CTC++ feature is activated in MTest (in
MTest_FindTool), MTest checks the validity of the
CTC installation (depending on the used code generator).
If any of the prerequisites is invalid a message is
provided to help solve the problem.- Batch report creation during batch run time
The batch report is continuously written after each
batch action now. This txt report is located beside
the catalog files.
If MATLAB (and thus MTest) freezes during a long batch
run (and must be shut down) you may check how far the
batch run was already completed (and restart the
remaining part).
If a problem occurred you can quickly recognize the
next not successfully finished batch action and
search (interactively) for the source of the problem.
This feature is added as one of the results of the
upcoming MTest classification.- Added check for length compliance of simulation output data
At the end of simulation (test execution) MTest checks
whether the length of input time and output vectors are
identical (and thus identical to input data).
This feature is added as one of the results of the
upcoming MTest classification.- RCT activated for non-MTCD test specifications too
RCT reports can be generated for non-MTCD specifications
as well (direct test data, CTE/Testona).- (Alpha) Controller Assessment utility functions
A first version of controller assessment functions
is included. They already proved their utility in
some projects.
Nevertheless, we are looking for feedback regarding
enhancements, missing features, incomplete features,
and further ideas.
BTW: The names of the functions may change with the
next major release (we are looking for even better
names).MTest v 3.4 (March 2016)
- Multi-parameter handling in MTCD actions enabled
An action line in MTCD can contain more than one named
parameter. This also includes MTCD functions, which can
be used with multiple parameters.
Named parameter settings from the test sequence
initialization are used even when a test group
initialization is present.- Batch GUI now shows sequence and test group name
The selection lists in the Batch GUI now show the name
of the test group and the test sequence directly.
This makes selecting special test groups and sequences
much easier.- Profiling of many batch actions saves quite some run time now
We profiled many of the batch actions and found some
lines of code which took too much run time. We improved
the implementation saving considerable run time.
This is most notable in projects with many sequences.- RCT project report included in report zip file
The report zip file (Documentation/Zip Test Catalogs and
Reports) now contains the RCT project report as well
(if available). The RCT xls(x) files are included as well.
The RCT project reports are saved in the same directory
(Test) as the test catalogs now (previously they were saved
directly in the project directory).- RCT report generation takes less time
The generation of the RCT reports could take quite a lot of
minutes for larger projects. Now it is back to seconds or
only a few minutes.- Extended interface analysis for signal parameters
The recognition of signal parameters in bus signals was
extended. The analysis of non-double signal data types
in busses has been improved.
After changing an interface the differences are displayed
much quicker for larger systems.- Enhanced usability for View Test Data and Outputs
The signal selection for viewing input and output signals
uses a standard list dialog now. This provides a
convenient way of selecting one or multiple signals.
Additionally, the axes of all plots are linked to each
other. This enables much easier synchronized zoom
functionality.
Up to 5 signals are plotted in one figure. Now all of
them are in one column (and not up to 2 columns as before).- Enhancements to assessments
The deactivated Active flag ('false' or 0) now disables
the assessment completely (the assessment does not run). If the CheckedRegion of a Passed assessment evaluation is
a vector containing only zeros, MTest adds an ATTENTION
note to the assessment result. This provides an additional
hint that your tolerances might be too large. The Assessment Analyzer now recognizes the simulation mode
in the assessment configuration (if one is configured). An implementation improvement in v 3.3 to handle check
region with mixed NaN/Inf boundary definitions in
MTa_compareSignals produced a longer run time than usual.
We improved the run time of the new code. Thus, regression
and back-2-back tests are quick again. Scalar triggered, complied, and checked region signals
are saved and postprocessed as full vectors (i.e. generation
of failed messages and visualization in Assessment Analyzer).- Evaluation author is taken from test group data
The author (aka tester) of the evaluation data is taken
from the data in the test group now (previously the
user name from the project data was used).- Always reload DD CodeCov data during long TL simulations
During TL simulations with code coverage enabled the
code coverage experiment data must be saved before code
gen and reloaded after code generation to enable aggregation
of code cov data. When code generation for one sequence
fails, MTest still reloads the saved code cov data to
enable the aggregation of the remaining code cov data.- Always clear output, eval, and local data before simulation
This clear step was already done - except if the code
generation in batch mode was not successful. Now the
clearing of these data files is still done (which provides
an additional feedback that something did not succeed
beside the screen messages).- Replaced backslash in link path of HTML reports and catalogs
A few links in HTML reports and catalogs could contain
backslahes instead of slashes. This could produce non
working links, when the reports are located on a server
drive (except for IE, which handles this). Now these
links are conform.- Attention
MES Tool users with dongle licenses may experience license
key problems after a recent update of the WibuKey dongle
driver. (In most cases, this update is related to the
installation of dSPACE TargetLink 4.1.)
WibuKey driver versions 6.31 and higher are affected. These
are distributed with applications ready for WibuKey/CodeMeter
licensing.
A fix is available upon request.MTest v 3.3 (Dezember 2015)
- Extended handling of special data types in test bed creation
The conversion of test input data according to original
data types in the test bed was extended.
Especially AliasTypes (as used with Embedded Coder) are
used now instead of the compiled data type as before.
The special compiled action data type for action trigger
signals is changed to double as well.- If/else triggered subsystems are handled as testobjects
Subsystems, which are triggered/activated by an if/else
block can be directly used as testobject. During
interface analysis the action trigger port is recognized.
During test bed generation the action trigger is driven
by an if-block, which can be switched on/off by the
respective input signal.
The automatic enhancement of such if/else triggered
subsystems from a TL subsystem if the interface does not
consist of TL in/outputs is not implemented yet.- Referenced model blocks as test objects
The blocks containing referenced models can be used as test
objects now. This is important if your referenced models
are not structured by subsystems (all functionality on
top level).
Current limitations: no SiL mode test bed for TL and EC
yet (we are working on it).- All HTML reports created in utf-8 for all characters
The report creation was changed to utf-8. This helps
especially in the preservation of chinese characters
(and probably many others not seen yet).- Helper function provided to switch MATLAB GUI to utf-8
MTest provides a function MTest_SetupUTF8Encoding to set up
the MATLAB environment to cope with utf-8 characters.
After calling the function any new (MTest) GUI item will
display the proper utf-8 characters.
See help of the function for more information.- Enhancements to assessments
During assessment generation the selection of the covered
requirements is enhanced by displaying the beginning of
the requirement description beside the requirement ID. Requirements and signals are sorted by ID / name in the
respective selection lists. The layout of the Assessment Analyzer was enhanced. The
plots use as much area as possible. The signals are plotted
using stairs. MTa_checkReaction now focuses on processing logical reaction
types. This improves understanding and facilitates a clearer
usage.
The reaction type (third input parameter) must be set
explicitly now (possible values are: alltrue/allfalse,
anytrue/anyfalse, notrue/nofalse). This provides a better
assurance of user defined assessments.
Older options are still available, but are not documented
any more and should be avoided in the future.
In some cases it is necessary to adapt already existing
assessments: when the reaction type is not defined (third
input parameter). You will be notified by an error during
first evaluation of this assessment.- Full review of the MTest Assessment Framework
In preparation for the ongoing MTest qualification the
MTest Assessment Framework was completely reviewed.
You may recognize an extended or easier to understand
help section. And yes, a few unit test cases to cover
rare cases had to be written.- In SL VnV Model Coverage no 'ConditionallyExecuteInputs'
MTest deactivates the model coverage configuration option
'ConditionallyExecuteInputs' when simulating the test bed
with activated model coverage.
This prevents the VnV toolbox from providing 'faulty'
coverage results when simultaneously dealing with switch
blocks and this activated option.- MTest projects also recognized when drive letter changed
When changing MTest projects or loading a project
the saved path of the project is even recognized
when the drive letter changed (only changed drive letter
is recognized).
This is often the case with version control systems,
where the "sandbox" may change drive letter.- Removed problem of test group limitation in catalog
Found the wrong search of test groups, which limited
the number of test groups in test catalogs to 99. Usage
of reworked list function (see below) provides enhanced
robustness.- New Model/Code Coverage Framework in MTest
A new model/code coverage framework for the handling
of the coverage data values was added. It is already
part of the release. Currently it is not activated
by default. It may be activated in MTest_FindTool.
Results from the new Coverage Framework are currently
only used in the RCT dashboard.- New internal methods for administrative info
Rework of methods for lists of test groups and test sequences.
Enables recovering from "unintended" deletion in file system
of test groups and test sequences.
The legacy testindex and Sequences vectors are no longer
needed (but still available). These vectors are updated
whenever a testobject (testindex vector) or test group
(Sequences) is selected.
Removed some internal legacy info, which is no longger
necessary (rb's). This leads to even shorter sequence
change times.MTest v 3.2 (September 2015)
- Extensions to RCT Framework
The work status names for test sequences and assessments
now have more meaningful names:
created Item created
reqchanged One of the referred requirements changed
described Descriptive data provided
inwork Item still in work
rejected Item rejected after review process
completed Item completed by tester and ready to use
reviewed Item passed the review process
In case of legacy status names a message is printed in the
MATLAB command window pointing to the affected artifact. Two metrics were added: assessment state monitor, and
test sequence state monitor. They are quantifications
of the status of all the assessments resp. test sequences.
These metrics have a value of 100% if all assessments/test
sequences are reviewed. Requirements can now be assigned to more than just one test object in
the subsystem column of the RSD. API call to generate the Project RCT report changed to:
MTr_RCT('action', 'generateProjectRCT')- Export test metrics into MQC format (MQC Exporter)
All metrics from RCT can be exported to MQC (MES Quality
Commander) format. Use Documentation/Generate MQC data.
The data contains the current state from the MTest project
(aka current revision).
Beside the metrics the current project structure is exported
to MQC format as well (to quickly generate the test project
structure in an MQC project).- Enhancement of test reports and catalogs design
The reports received further design enhancements.- MTCD test group init section in test sequence report
The init part of MTCD test group is documented in the test
sequence report beside the test sequence initialization
and action. Now all parts of MTCD definition are shown in
the test sequence report.- Added MTCD function pulsewave()
MTCD now provides an action function for generating
square/rectangle wave signals. See MTest User Guide
for examples.- Clear of MTCD import cache file
To ensure up-to-date parameter values during MTCD import
MTest clears the internal MTCD import cache file whenever
an MTCD import is performed interactively (i.e. via menu)
or during batch mode each time a test group is entered.- Documentation of parameter variation in test report
Changes in the used parameter values per sequence (compared
to standard parameter settings) are documented in the
test sequence report at the beginning of the test input
section.
For each changed parameter the used/changed value is
shown together with the standard/default value.
Parameter structures are analyzed down to changed scalar
values and only the changed values are documented.- Documentation of used parameters for each test object
The (default) parameters of the current test object are
documented in a separate Parameter Report. This report is
saved in the test object directory. A link to the parameter
report can be found in each test sequence report at the
beginning of the input section.- Added initial support for TL 4.0
The available functionality also works with TL 4.0.
Further features of TL 4.0 will be supported in the next
release (aka extended TL AUTOSAR support).- Enhancements to user experience
After selecting a test object the first test group (if
available) is selected automatically. New GUI menu entry to open configuration of signal tolerances
for regression and back-2-back testing
Evaluation/Configure Signal Tolerance
If the signal tolerance config file does not exist, it is
generated before opening it. The function to clean up internal MTest data is accessible by
the new GUI menu entry: Help/Clear data
Using this call all or selected data files of the current
MTest project or test model can be deleted or reset. An all
option is available on the command line to be used in batch
scripts: MTest_cleanDirectoriesf('all', 'project')- More ready to run examples added for evaluation configuration
The examples for evaluation configuration in
Evaluation/Configure Test Evaluation (MTest_SystemOptions)
are extended for more assessment variants and additional
back-to-back testing.- Name and description of MTest project checked
The name and description of MTest project is checked for
invalid characters. The project name is used for the test
catalog file name and must contain only characters supported
for file names.
The project description should not contain string delimiter.- No model coverage in Embedded Coder SiL and PiL simulation
Disabled possibility to perform model coverage in Embedded
Coder SiL (and PiL) simulation. Model coverage is only for
MiL mode.- Fixed rare problem of not complete TL code coverage aggregation
When running MiL and SiL simulation in batch for the
very first time and collecting TL code coverage, the
aggregation during this batch run did not work. From the
second run on the results were correct. This first run
problem is solved now.- Fixed problem during regeneration of test beds in batch mode
The regeneration of test beds in batch when the test beds
were not there at the beginning of the batch run produced
a failure for the second test bed. This works now as
expected from the beginning.- Removed bug in Edit/View Test Data (and Outputs)
There was a (matrix concatenation) bug for rare cases in
View Test Data (and Outputs). This part was refactored.
In very special cases a few outputs were not available for
selection in the signal selection list.- Global simulation parameter loaded before code generation
The global simulation parameters are loaded before code
generation in batch mode (and not only before simulation).- Further extension of AutoPilot demo example
Added separate model for R2014b.
The reporting of run-time errors was extended. The checking
for simulation problems is easier now.
Use of new calls to RCT framework.- Added support for fixdt data types in test bed generation
Instead of the compiled data type for fixdt DataTypes the
textual data type (fixdt(a,b,c)) is used. This produces easier
to handle test bed for Embedded Coder models.- Changed generation of Embedded Coder test beds
The test bed for Embedded Coder (EC) models was changed.
MTest keeps separated test beds for MiL and SiL mode (the
online change of the MiL/SiL test object produced quite
a problem with V&V model coverage in newer ML versions).
Now the change between MiL and SiL test bed is done by
switching between the respective test bed models.
When switching from older MTest versions with EC models,
please regenerate your test bed once to use the new
mechanism (by using regenerate test beds in batch for
instance).- (Alpha) Active referenced config set converted to active config set
If a referenced config set is used in the model (and the
active one), this referenced config set is copied to the test
bed as any other config set (long time existing feature).
Afterwards, the config set which is referenced is copied
to the test bed and activated. In this way parameters
of the configuration can be changed in the test bed.MTest v 3.1 (June 2015)
- Enhanced structure of assessments and better template generation
The suggested assessment structure was extended. Now two
sections are available: Data of assessment and Execution
information of assessment.
Using the data section a name and a description can be given
for each assessment (and a few more attributes). This
information can be defined long before the assessment can be
executed and it extends the documentation of assessment
considerably.
The execution section contains all the information for the
execution of the assessment. Some new fields are returned to
the MTest assessment framework for further processing (for
instance in the Assessment Inspector).
The assessment template generator (Evaluation/New Assessment
Template) provides an interactive way to gather the main
information needed for the template (which requirements are
assessed, ID of assessment and the used signals). The template
uses the suggested assessment structure and a lot of
additional information useful during assessment editing.
Of course, the previous assessment format is still fully supported.
However, quite a few of the new features (Inspector) are not
possible with the old assessment format.- Added menu entry to Debug Assessments
Using Evaluation/Debug Assessments the defined assessments
can be accessed and run on the current test sequence directly.
Quite some information about the assessment and test sequence
is output in the MATLAB command window. The Assessment
Inspector is opened here as well.- (Alpha) Added Assessment Analyzer (Inspector)
Provides information and signal graphics for one assessment run
on one test sequence. Main use is for reviewing and debugging
assessments (without writing an own plot script).
Try Evaluation/Debug Assessments and have a look. Especially
inspecting the triggered and complied regions is always
interesting and provides in-depth insight. Reviewing assessments
is now much easier.
Please be aware, this is a first alpha version - already
helpful. More is on the way.- Extension of RCT Framework
RCT not only reads MTCD test specifications but also all other
types of test specifications MTest can work with.
The updated assessment format is included in RCT.
The labels of the RCT report and metrics were reworked (Test
Sequences instead of Tests).
The boundary of the target values of metrics is included in
target range (>= instead of >).- GUI and Batch inclusion of CTC++ code coverage
The CTC++ can be configured in the GUI (Execution/Execution
options). The code coverage level selection contains beside
TL code coverage also the CTC++ code coverage.
The same selection of code coverage levels can be done in
batch (GUI) mode.
Changes in switching CTC++ on and off (developed together with
CTC++ developers) provide better support for 64bit with Embedded
Coder (tested with R2013b and R2014b).- (Beta) Conversion from MTCD to CTE/TESTONA
Extended support for the conversion of MTCD test specification
to CTE/TESTONA test specifications.
Using MTest_ConvertMTCD2CTE complete MTCD test specifications
can be transfered to an identical CTE/TESTONA file (as far as
possible).
Many of the asynchronous features of MTCD (ramps outside test
steps and so on) are transferred by conversion to multiple test
steps.- Extended checking for loaded/opened model
Before analyze interface and before testbed generation MTest
checks if the model is loaded/opened (this is done after model
selection, but the model could be closed by the tester
inadvertently). If the model is not open it is loaded in the
background.- Automatic interface analysis before new test bed/effective test interface
Before a new test bed is (re)generated the interface of the
test object is analyzed automatically. Thus, if the interface
changed, these changes are taken into account during test bed
generation.
The same is done before accessing the effective test interface.
This ensures a more consistent workflow in the case of changing
test object interface.
Please remember to adapt these changes in your test specification
and assessments.- Reporting of changes in interface of test object
Changes in the interface of the test object (Analyse interface)
are reported in the MATLAB command window (and in a message box
for interactive Analyze Interface).- Separated MATLAB path setting and Java path setting
A separate function for adding MTest paths to MATLAB path and some
Java classes and lib dirs to Java Class Path was created.
Using mtest_paths the MATLAB and Java path can be extended long
before MTest is started.
This can and should be used in an integrated development environment,
where multiple applications extend the Java class path (including
the MES tools) and use java objects.
Why this is necessary: After changing the (dynamic) java class
path MATLAB clears java and thus all Java objects. In this way you
destroy the MES license handle object (and similar java objects
of other applications).
What this means: if you are using multiple tools then set all path
extensions at the beginning of your MATLAB session at once.
For MTest use: mtest_paths('add')
Afterwards start the tool(s). MTest does not extend the java path
again (and thus the java objects survive).
If you want to only use MTest, just start MTest in the usual way.
MTest checks at startup if all necessary paths are set (and if
they are set, nothing is done).- Further extension of AutoPilot demo example
The demo example contains more status checks, more status
information, and an extended result output.
A few more feature possibilities are included in the demo.
It can also be used for MATLAB versions other than R2011b or
and R2013b (using the models of the next lower version).MTest v 3.0 (May 2015)
- Significant speed-up of runtime
We revised some internal processes so that MTest is much faster
now.- Rework of design of test reports and catalogs
The test reports and catalogs received a design overhaul.
We like it a lot more than before.- Extension of RCT framework
Now, the report generation is fully integrated into the batch test.
(GUI and MTest_Batch or MTest_BatchProject).
RCT now contains a completely revised set of (standard) metrics.
We performed a major overhaul of all report appearances.
The handling of a single RSD is supported.
An auto-testability functionality for automated filtering of
testable requirements is available.
The testers review of the imported requirements is called from the
GUI using Requirements -> Review Requirements / Manage Testability- Extension of the generated MTCD template for test specification
The default parameter definition in the test group (first
row) now contains the default parameter values (if known
by test setup).
The MTCD template contains an additional column
to define/track the status of the respective test sequence
definition. This status information is used (and aggregated)
in RCT.- Extension of conversion of test sequences from signal builder to MTCD
The extended features of MTCD (default parameters, additional
column for setting the RCT status) are available in converted
signal builder MTCD files as well.- License mechanism included and enabled
The Flex license mechanism (already known from MXAM and M-XRAY)
is included and enabled.
Please have a look to readme_installation.txt in the main MTest
directory for installation instructions for the different
license variants (dongle, floating, ...).
This is the same mechanism as in MXAM 3.x and M-XRAY 2.x.
Info on the used license can be called by
MTest GUI/Help/View License Info (output in command window).- Added menu entry to Open Assessments
Using Evaluation/Open Assessments the defined assessments
can be accessed (aka opened for view anmd edit) directly.- Extended AutoPilot demo example with separate SL/TL/EC models
Extended demo example which contains the same autopilot
example as Simulink (SL) model, as a TargetLink (TL) model,
and an Embedded Coder (EC) model. The functionality is identical
in all model variants.
This complete demo project is available in R2011b/TargetLink 3.4
and R2013b/TargetLink 3.5 variants (ready to run).
The complete multi-platform test project can be generated using
the demo file: MTest_Demo_SysTest_Autopilot('multi').
Yes, it contains quite a few Failed.- Deprecated old way of configuring assessments
The configuration of assessments is done in
Evaluation/Configure Test Evaluation.
There was a possibility in Documentation/Configure Test
Sequence Report a long time ago (which is still working as
backup). If you are still using this old way, you will
receive a deprecated message in the MATLAB command window.
Please change this old configuration to the correct way.- Global simulation options also used for code generation
The globally defined simulation options are now loaded before
code generation (and not only before simulation) in batch mode.
Thus, defined continuous solvers and more are available during
parameter checking of the code generator (i.e. TL).- MTest_Batch handles multiple models now in one call
The main batch MTest_Batchfunction was extended to handle
multiple models in one call. See help MTest_Batch for all
the details.
BTW, please use MTest_BatchProject to run complete projects.- MTest_Batch produces batch summary information now
The batch summary output known from Batch GUI runs is now
also available in calls to MTest_Batch (and thus
MTest_BatchProject as well).
the output of the information was slightly reformatted
for better usability.
An overall batch run result is displayed at the end for a
quick indication, if all the batch steps were successful
or not.- Renamed Clean code option in Batch GUI to SiL/PiL clean code
This clean code option in batch test dialog is used for SiL
and Pil at the same time. This is now reflected in the label
inside the dialog.- Added support for Testona 4.x
MTest now supports TESTONA 4.x (the new name of the tool
formerly known as CTE XL).
To use Testona (instead of CTE XL 3.x) define the CTE
directory in Project settings using the path to the Testona
installation directory.
When using Testona, your "old" CTE files must be transfered
to the new format once (new extension .testona and very small
change in internal XML format). Just navigate in Testona to
the .cte files, open them and save as .testona. Afterwards
they can directly be used in MTest.
BTW, CTE/XL 3.x is still supported - nothing changed there.- Convenient access to release notes
The current release notes can now be called conveniently
from the GUI using Help/View Release Notes- Renamed verify utility functions to MTest name space
The utility functions were renamed from mes_* to MTest_*.
mes_verifyInput is now MTest_verifyInput and
mes_getVerifiedInput is now MTest_getVerifiedInput.
The old names are moved to AddOn/obsolete for a few releases.- (Beta) Adapt relative path settings in TL DD search paths
Inside the TL DD the 'IncludeFileSearchPath' and
'SourceFileSearchPath' definitions can include relative path
settings. When the DD is copied from model to test bed these
relative path settings are now adapted (extended by the
relative path from test bed directory to model directory).- (Beta) Extended adjustment of top level TL Function block step function
The adjustment is only done, when no GLOBAL_FCN definition
is found inside step function class (also definitions in a
FunctionClassGroup are taken as valid global definition.)
In some cases the definition of TL GLOBAL_FCN is not in
/Pool/FunctionClasses/GLOBAL_FCN. For this case MTest looks
for the defined path of GLOBAL_FCN and uses this special path
for the adjustment of TL step function class for top level
TL Function blocks. If multiple definitions of GLOBAL_FCN
exist, the first one is used- (Beta) New interface description routines
See AddOn/MTS for more details.- Extended CTC code coverage integration
The generated CTC code coverage reports are linked in and can
be viewed directly from the test sequence report (similar to
model and TL code coverage reports).
The integration of CTC was extended for better robustness
and usability (but seldom some problems with EC still occur).- Correction of sin signal generation from MTCD
The sin signal starts now correctly at the very beginning of
the test step. Previously this start of the sin signal was
one simulation step too late.
This produced very small deviation when comparing two
sequences with identically defined sin but different length
of sin signal (now same signal form for same period duration).- Corrected reset of previous assessment results
When using MTa_loadParameters4Assmnt outside of the intended
assesment routine (i.e. on the MATLAB command line or in own
scripts) the current assessment results of this test sequence
were reset.
Now the routine may be used (still not intended) without
reseting assessment results (which is done at another place).- Project name may contain : now
Catalog files were not created with the correct name, if
the project name contained : (which is replaced by _ now).- Extension of cleanup functionality of MTest data
The different reset functions were reworked and reset data
as expected. The Testsequence not defined message is fixed.- Extended internal method to cache administrative info
Cache of model and project information is active.