TestNG Documentation

TestNG Documentation
TestNG Documentation
7.9.0
Table Of Contents
1. Welcome to TestNG
1.1. Requirements
1.2. Mailing-lists
1.3. Locations of the projects
1.4. Bug reports
1.5. License
2. Download
2.1. Current Release Version
2.1.1. Maven
2.1.2. Gradle
2.1.3. Snapshots
2.1.4. Eclipse plug-in
2.1.5. Build TestNG from source code
2.1.6. Build the TestNG Eclipse Plugin from source code
3. TestNG Documentation
3.1. Introduction
3.2. Annotations
3.3.
testng.xml
3.4. Running TestNG
3.4.1. Command Line Parameters
3.4.2. System properties
3.4.3. Running tests from within a test jar
3.5. Test methods, Test classes and Test groups
3.5.1. Test methods
3.5.2. Test groups
Method groups
3.5.3. Groups of groups
3.5.4. Exclusion groups
3.5.5. Partial groups
3.6. Parameters
3.6.1. Parameters from testng.xml
3.6.2. Parameters with DataProviders
3.6.3. Retries and data providers
3.6.4. Controlling ThreadPool Usage
3.6.5. Parameters from System Properties
Parameters in reports
3.7. Dependencies
3.7.1. Dependencies with annotations
3.7.2. Dependencies in XML
3.8. Factories
3.9. Class level annotations
3.10. Ignoring tests
3.11. Parallelism and time-outs
3.11.1. Parallel suites
3.11.2. Parallel tests, classes and methods
3.12. Rerunning failed tests
3.13. JUnit tests
3.14. BeanShell and advanced group selection
3.14.1. Annotation Transformers
3.14.2. Via xml suite file
3.14.3. Via command line arguments
3.14.4. Via your code
3.15. Method Interceptors
3.16. Interceptors for Data Providers
3.17. TestNG Listeners
3.17.1. Listener list
3.17.2. Specifying listeners with testng.xml or in Java
3.17.3. Specifying listeners with ServiceLoader
3.17.4. Ordering listeners in TestNG
3.18. Dependency injection
3.18.1. Native dependency injection
3.18.2. Guice dependency injection
3.19. Listening to TestNG lifecycle events
3.20. Listening to method invocations
3.21. Listening to configuration invocations
3.22. Listening to class level invocations
3.23. Listening to data provider invocations
3.24. Listening to Suite level invocations
3.25. Overriding test methods
3.26. Overriding configuration methods
3.27. Altering suites (or) tests
4. Test results
4.1. Success, failure and assert
4.2. Logging and results
4.3. Logging Listeners
4.4. Logging Reporters
4.5. JUnit Reports
4.6. Reporter API
4.7. XML Reports
4.8. TestNG Exit Codes
5. YAML
6. Dry Run for your tests
7. JVM Arguments in TestNG
8. Logging framework integration in TestNG
9. Migrating from JUnit
9.1. Using Eclipse
9.2. Asserts
9.3. Running JUnit Tests
9.4. Related reading
10. Javadocs
11. Running TestNG
11.1. Eclipse plug-in
11.2. TestNG IDEA Plug-in
11.3. Using TestNG with Gradle
11.3.1. Specifying your Gradle build file
11.4. Using TestNG with Maven
11.4.1. Specifying your pom.xml
11.4.2. Sample reports
11.5. TestNG Ant Task
11.6. Running TestNG programmatically
12. MORE TESTNG READING
12.1. Articles and presentations
12.1.1. English
12.1.2. French
12.1.3. German
12.2. CONTRIBUTORS
12.3. SPECIAL THANKS
12.4. TESTIMONIES
13. The TestNG TextBook: Next Generation Java Testing
13.1. Book Description
13.2. About the Author
14. ReportNG - A reporting tool
15. GitHub Repository
15.1. TestNG Core
15.2. TestNG Documentation
15.3. TestNG Eclipse plugin
15.4. ReportNG - A reporting tool
1. Welcome to TestNG
TestNG is a testing framework inspired from JUnit and NUnit but introducing some new functionalities that make it more powerful and easier to use, such as:
Annotations.
Run your tests in arbitrarily big thread pools with various policies available (all methods in their own thread, one thread per test class, etc…​).
Test that your code is multithread safe.
Flexible test configuration.
Support for data-driven testing (with @DataProvider).
Support for parameters.
Powerful execution model (no more TestSuite).
Supported by a variety of tools and plug-ins (Eclipse, IDEA, Maven, etc…​).
Embeds BeanShell for further flexibility.
Default JDK functions for runtime and logging (no dependencies).
Dependent methods for application server testing.
TestNG is designed to cover all categories of tests: unit, functional, end-to-end, integration, etc…​
I started TestNG out of frustration for some JUnit deficiencies which I have documented on my weblog here and here Reading these entries might give you a better idea of the goal I am trying to achieve with TestNG. You can also check out a quick
overview of the main features
and an
article
describing a very concrete example where the combined use of several TestNG’s features provides for a very intuitive and maintainable testing design.
Here is a very simple test:
package example1;

import org.testng.annotations.*;

public class SimpleTest {

@BeforeClass
public void setUp() {
// code that will be invoked when this test is instantiated

@Test(groups = {"fast"})
public void aFastTest() {
System.out.println("Fast test");

@Test(groups = {"slow"})
public void aSlowTest() {
System.out.println("Slow test");

The method
setUp()
will be invoked after the test class has been built and before any test method is run. In this example, we will be running the group fast, so
aFastTest()
will be invoked while
aSlowTest()
will be skipped.
Things to note:
No need to extend a class or implement an interface.
Even though the example above uses the JUnit conventions, our methods can be called any name you like, it’s the annotations that tell TestNG what they are.
A test method can belong to one or several groups.
Once you have compiled your test class into the build directory, you can invoke your test with the command line, an ant task (shown below) or an XML file:

classname="org.testng.TestNGAntTask"/>

Use ant to invoke it:
c:> ant
Buildfile: build.xml

test:
[testng] Fast test
[testng] ===============================================
[testng] Suite for Command line test
[testng] Total tests run: 1, Failures: 0, Skips: 0
[testng] ===============================================

BUILD SUCCESSFUL
Total time: 4 seconds
Then you can browse the result of your tests:
start test-output\index.html (on Windows)
1.1. Requirements
TestNG Upto v7.5: JDK 8.
TestNG v7.6.0 and above: JDK 11 or higher.
1.2. Mailing-lists
The users mailing-list can be found on
Google Groups
If you are interested in working on TestNG itself, join the
developer mailing-list
1.3. Locations of the projects
If you are interested in contributing to TestNG or one of the IDE plug-ins, you will find them in the following locations:
TestNG
Eclipse plug-in
IDEA IntelliJ plug-in
Visual Studio Code plugin-in
1.4. Bug reports
If you think you found a bug, here is how to report it:
Create a small project that will allow us to reproduce this bug. In most cases, one or two Java source files and a testng.xml file should be sufficient. Then you can either zip it and email it to the
testng-dev mailing-list
or make it available on an open source hosting site, such as
github
and email testng-dev so we know about it. Please make sure that this project is self contained so that we can build it right away (remove the dependencies on external or proprietary frameworks, etc…​).
If the bug you observed is on the Eclipse plug-in, make sure your sample project contains the
.project
and
.classpath
files.
File a bug.
For more information, you can either
download TestNG
, read the manual or browse the links at the top.
1.5. License
Apache 2.0
2. Download
2.1. Current Release Version
2.1.1. Maven

org.testng
testng
7.9.0
test

2.1.2. Gradle
dependencies {
testCompile 'org.testng:testng:7.9.0'
2.1.3. Snapshots
TestNG automatically uploads
snapshots to Sonatype
which you can access by adding the following repository:
repositories {
maven {
url 'https://oss.sonatype.org/content/repositories/snapshots'
2.1.4. Eclipse plug-in
See
the related documentation
2.1.5. Build TestNG from source code
TestNG is also hosted on GitHub, where you can download the source and build the distribution yourself:
$ git clone https://github.com/testng-team/testng.git
$ cd testng
$ ./gradlew build
You will then find the jar file in the
testng/build/libs
directory
Some useful tasks:
./gradlew tasks
- to see what all tasks are available
./gradlew test
- to kick off the tests (incremental build)
./gradlew clean test
- if you want to have a clean build.
./gradlew autostyleApply
- Applies code formatting steps to sourcecode in-place.
./gradlew autostyleCheck
- Checks that sourcecode satisfies formatting steps.
./gradlew check
- Runs the below checks:
"Applies code formatting steps to sourcecode in-place." and
"Checks that sourcecode satisfies formatting steps.".
2.1.6. Build the TestNG Eclipse Plugin from source code
TestNG Eclipse Plugin is
hosted on GitHub
, you can download the source code and
build by ourselves
3. TestNG Documentation
3.1. Introduction
TestNG is a testing framework designed to simplify a broad range of testing needs, from unit testing (testing a class in isolation of the others) to integration testing (testing entire systems made of several classes, several packages and even several external frameworks, such as application servers).
Writing a test is typically a three-step process:
Write the business logic of your test and insert
TestNG annotations
in your code.
Add the information about your test (e.g. the class name, the groups you wish to run, etc…​) in a testng.xml file or in
build.xml
Run TestNG
You can find a quick example on the
Welcome page
The concepts used in this documentation are as follows:
A suite is represented by one XML file. It can contain one or more tests and is defined by the

tag.
A test is represented by and can contain one or more TestNG classes.
A TestNG class is a Java class that contains at least one TestNG annotation. It is represented by the tag and can contain one or more test methods.
A test method is a Java method annotated by
@Test
in your source.
A TestNG test can be configured by
@BeforeXXX
and
@AfterXXX
annotations which allows to perform some Java logic before and after a certain point, these points being either of the items listed above.
The rest of this manual will explain the following:
A list of all the annotations with a brief explanation. This will give you an idea of the various functionalities offered by TestNG but you will probably want to consult the section dedicated to each of these annotations to learn the details.
A description of the testng.xml file, its syntax and what you can specify in it.
A detailed list of the various features and how to use them with a combination of annotations and testng.xml.
3.2. Annotations
Type
Annotations
Description
Annotation
@BeforeSuite
@AfterSuite
@BeforeTest
@AfterTest
@BeforeGroups
@AfterGroups
@BeforeClass
@AfterClass
@BeforeMethod
@AfterMethod
Configuration information for a TestNG class:
@BeforeSuite
: The annotated method will be run before all tests in this suite have run.
@AfterSuite
: The annotated method will be run after all tests in this suite have run.
@BeforeTest
: The annotated method will be run before any test method belonging to the classes inside the tag is run.
@AfterTest
: The annotated method will be run after all the test methods belonging to the classes inside the tag have run.
@BeforeGroups
: The list of groups that this configuration method will run before. This method is guaranteed to run shortly before the first test method that belongs to any of these groups is invoked.
@AfterGroups
: The list of groups that this configuration method will run after. This method is guaranteed to run shortly after the last test method that belongs to any of these groups is invoked.
@BeforeClass
: The annotated method will be run before the first test method in the current class is invoked.
@AfterClass
: The annotated method will be run after all the test methods in the current class have been run.
@BeforeMethod
: The annotated method will be run before each test method.
@AfterMethod
: The annotated method will be run after each test method.
Behaviour of annotations in superclass of a TestNG class
The annotations above will also be honored (inherited) when placed on a superclass of a TestNG class. This is useful for example to centralize test setup for multiple test classes in a common superclass.
In that case, TestNG guarantees that the "@Before" methods are executed in inheritance order (highest superclass first, then going down the inheritance chain), and the "@After" methods in reverse order (going up the inheritance chain).
Annotation Attributes
alwaysRun
For before methods (beforeSuite, beforeTest, beforeTestClass and beforeTestMethod, but not beforeGroups): If set to true, this configuration method will be run regardless of what groups it belongs to.
For after methods (afterSuite, afterClass, …​): If set to true, this configuration method will be run even if one or more methods invoked previously failed or was skipped.
dependsOnGroups
The list of groups this method depends on.
dependsOnMethods
The list of methods this method depends on.
enabled
Whether methods on this class/method are enabled.
groups
The list of groups this class/method belongs to.
inheritGroups
If true, this method will belong to groups specified in the @Test annotation at the class level.
onlyForGroups
Only for @BeforeMethod and @AfterMethod. If specified, then this setup/teardown method will only be invoked if the corresponding test method belongs to one of the listed groups.
Annotation
@DataProvider
Marks a method as supplying data for a test method. The annotated method must return an
Object[][]
where each
Object[]
can be assigned the parameter list of the test method. The @Test method that wants to receive data from this
DataProvider
needs to use a dataProvider name equals to the name of this annotation.
Annotation Attributes
name
The name of this data provider. If it’s not supplied, the name of this data provider will automatically be set to the name of the method.
parallel
If set to true, tests generated using this data provider are run in parallel. Default value is false.
Annotation
@Factory
Marks a method as a factory that returns objects that will be used by TestNG as Test classes. The method must return
Object[]
Annotation
@Listeners
Defines listeners on a test class.
Annotation Attributes
value
An array of classes that extend org.testng.ITestNGListener.
Annotation
@Parameters
Describes how to pass parameters to a @Test method.
Annotation Attributes
value
The list of variables used to fill the parameters of this method.
Annotation
@Test
Marks a class or a method as part of the test.
Annotation Attributes
alwaysRun
If set to true, this test method will always be run even if it depends on a method that failed.
dataProvider
The name of the data provider for this test method.
dataProviderClass
The class where to look for the data provider. If not specified, the data provider will be looked on the class of the current test method or one of its base classes. If this attribute is specified, the data provider method needs to be static on the specified class.
dependsOnGroups
The list of groups this method depends on.
dependsOnMethods
The list of methods this method depends on.
description
The description for this method.
enabled
Whether methods on this class/method are enabled.
expectedExceptions
The list of exceptions that a test method is expected to throw. If no exception or a different than one on this list is thrown, this test will be marked a failure.
groups
The list of groups this class/method belongs to.
invocationCount
The number of times this method should be invoked.
invocationTimeOut
The maximum number of milliseconds this test should take for the cumulated time of all the invocationcounts. This attribute will be ignored if invocationCount is not specified.
priority
The priority for this test method. Lower priorities will be scheduled first.
successPercentage
The percentage of success expected from this method
singleThreaded
If set to true, all the methods on this test class are guaranteed to run in the same thread, even if the tests are currently being run with
parallel="methods"
. This attribute can only be used at the class level and it will be ignored if used at the method level. Note: this attribute used to be called sequential (now deprecated).
timeOut
The maximum number of milliseconds this test should take.
threadPoolSize
The size of the thread pool for this method. The method will be invoked from multiple threads as specified by invocationCount.
NOTE: this attribute is ignored if invocationCount is not specified
3.3.
testng.xml
You can invoke TestNG in several different ways:
With a
testng.xml
file
With
ant
From the command line
This section describes the format of
testng.xml
(you will find documentation on ant and the command line below).
The current DTD for testng.xml can be found on the main Web site:
testng-1.0.dtd
Here is an example testng.xml file:

You can specify package names instead of class names:

In this example, TestNG will look at all the classes in the package
test.sample
and will retain only classes that have TestNG annotations.
You can also specify groups and methods to be included and excluded:

You can also define new groups inside
testng.xml
and specify additional details in attributes, such as whether to run the tests in parallel, how many threads to use, whether you are running JUnit tests, etc…​
By default, TestNG will run your tests in the order they are found in the XML file. If you want the classes and methods listed in this file to be run in an unpredictable order, set the
preserve-order
attribute to false.











Please see the DTD for a complete list of the features, or read on.
3.4. Running TestNG
TestNG can be invoked in different ways:
Command line
ant
Eclipse
IntelliJ’s IDEA
This section only explains how to invoke TestNG from the command line.
Please click on one of the links above if you are interested in one of the other ways.
Assuming that you have TestNG in your class path, the simplest way to invoke TestNG is as follows:
java org.testng.TestNG testng1.xml [testng2.xml testng3.xml ...]
You need to specify at least one XML file describing the TestNG suite you are trying to run.
Additionally, the following command-line switches are available:
3.4.1. Command Line Parameters
Option
Argument
Documentation
-configfailurepolicy
skip
continue
Whether TestNG should continue to execute the remaining tests in the suite or skip them if an @Before* method fails. Default behavior is
skip
-d
A directory
The directory where the reports will be generated (defaults to
test-output
).
-dataproviderthreadcount
The default number of threads to use for data providers when running tests in parallel.
This sets the default maximum number of threads to use for data providers when running tests in parallel. It will only take effect if the parallel mode has been selected (for example, with the
-parallel
option). This can be overridden in the suite definition.
-excludegroups
A comma-separated list of groups.
The list of groups you want to be excluded from this run.
-groups
A comma-separated list of groups.
The list of groups you want to run (e.g. "windows,linux,regression").
-listener
A comma-separated list of Java classes that can be found on your classpath.
Lets you specify your own test listeners. The classes need to implement
org.testng.ITestListener
-usedefaultlisteners
true
false
Whether to use the default listeners
-methods
A comma separated list of fully qualified class name and method. For example
com.example.Foo.f1,com.example.Bar.f2
Lets you specify individual methods to run.
-methodselectors
A comma-separated list of Java classes and method priorities that define method selectors.
Lets you specify method selectors on the command line. For example:
com.example.Selector1:3,com.example.Selector2:2
-parallel
methods,tests,classes
If specified, sets the default mechanism used to determine how to use parallel threads when running tests. If not set, default mechanism is not to use parallel threads at all. This can be overridden in the suite definition.
-reporter
The extended configuration for a custom report listener.
Similar to the
-listener
option, except that it allows the configuration of JavaBeans-style properties on the reporter instance.
Example:
-reporter com.test.MyReporter:methodFilter=
insert
,enableFiltering=true
You can have as many occurrences of this option, one for each reporter that needs to be added.
-sourcedir
A semicolon separated list of directories.
The directories where your javadoc annotated test sources are. This option is only necessary if you are using javadoc type annotations. (e.g.
src/test
or
src/test/org/testng/eclipse-plugin;src/test/org/testng/testng
).
-suitename
The default name to use for a test suite.
This specifies the suite name for a test suite defined on the command line. This option is ignored if the
suite.xml
file or the source code specifies a different suite name. It is possible to create a suite name with spaces in it if you surround it with double-quotes
"like this"
-testclass
A comma-separated list of classes that can be found in your classpath.
A list of class files separated by commas (e.g.
org.foo.Test1,org.foo.test2
).
-testjar
A jar file.
Specifies a jar file that contains test classes. If a
testng.xml
file is found at the root of that jar file, it will be used, otherwise, all the test classes found in this jar file will be considered test classes.
-testname
The default name to use for a test.
This specifies the name for a test defined on the command line. This option is ignored if the suite.xml file or the source code specifies a different test name. It is possible to create a test name with spaces in it if you surround it with double-quotes
"like this"
-testnames
A comma separated list of test names.
Only tests defined in a tag matching one of these names will be run.
-testrunfactory
A Java classes that can be found on your classpath.
Lets you specify your own test runners. The class needs to implement
org.testng.ITestRunnerFactory
-threadcount
The default number of threads to use when running tests in parallel.
This sets the default maximum number of threads to use for running tests in parallel. It will only take effect if the parallel mode has been selected (for example, with the -parallel option). This can be overridden in the suite definition.
-xmlpathinjar
The path of the XML file inside the jar file.
This attribute should contain the path to a valid XML file inside the test jar (e.g.
resources/testng.xml
). The default is
testng.xml
, which means a file called
testng.xml
at the root of the jar file. This option will be ignored unless
-testjar
is specified.
-shareThreadPoolForDataProviders
true
false
Indicates if TestNG should use a global shared thread-pool (at suite level) for running data driven tests. TestNG will consider the value set for the configuration parameter
-dataproviderthreadcount
as the size of the thread pool.
-useGlobalThreadPool
true
false
Indicates if TestNG should use a global shared thread-pool (at suite level) for running regular and data driven tests. TestNG will consider the value set for the configuration parameter
-threadcount
as the size of the thread pool.
-log
(or)
-verbose
a valid log level
Level of verbosity to be used when logging messages.
-junit
true
false
Should TestNG run in JUnit mode.
-mixed
true
false
Mixed mode - autodetect the type of current test and run it with appropriate runner.
-objectfactory
A string that represents a fully qualified class name.
Fully qualified class name that implements
org.testng.ITestObjectFactory
which can be used to create test class and listener instances.
-ignoreMissedTestNames
true
false
Ignore missed test names given by
-testnames
and continue to run existing tests, if any.
-skipfailedinvocationcounts
true
false
Should TestNG skip failed invocation counts for data driven tests and tests driven by invocation counts.
-testRunFactory
A string that represents a fully qualified class name.
Fully qualified class name that implements
org.testng.ITestRunnerFactory
which can be used to create custom test runners for running tests.
-suitethreadpoolsize
An integer value that represents the thread pool size. When not specified, defaults to
Size of the thread pool to use to run suites.
-randomizesuites
true
false
Whether to run suites in same order as specified in XML or not.
-alwaysrunlisteners
true
false
Should method invocation listeners be run even for skipped methods.
-dependencyinjectorfactory
A string that represents a fully qualified class name.
Fully qualified class name that implements org.testng.IInjectorFactory which can be used to handle with dependency injection.
`-failwheneverythingskipped'
true
false
Should TestNG fail execution if all tests were skipped and nothing was run.
-spilistenerstoskip
A comma separated string that represents a fully qualified class name(s).
Comma separated fully qualified class names of listeners that should be skipped from being wired in via Service Loaders.
-overrideincludedmethods
true
false
Should TestNG exclude explicitly included test methods if they belong to any excluded groups as defined in the suite xml file.
-includeAllDataDrivenTestsWhenSkipping
true
false
Should TestNG report all iterations of a data driven test as individual skips, in-case of upstream failures.
-propagateDataProviderFailureAsTestFailure
true
false
Should TestNG consider failures in Data Providers as test failures.
-generateResultsPerSuite
true
false
Should TestNG generate results on a per suite basis by creating a sub directory for each suite and dumping results into it.
-shareThreadPoolForDataProviders
true
false
Should TestNG use a global Shared ThreadPool (At suite level) for running data providers.
-useGlobalThreadPool
true
false
Should TestNG use a global Shared ThreadPool (At suite level) for running regular and data driven tests.
This documentation can be obtained by invoking TestNG without any arguments.
You can also put the command line switches in a text file, say
c:\command.txt
, and tell TestNG to use that file to retrieve its parameters:
C:> more c:\command.txt
-d test-output testng.xml
C:> java org.testng.TestNG @c:\command.txt
Additionally, TestNG can be passed properties on the command line of the Java Virtual Machine, for example
java -Dtestng.test.classpath="c:/build;c:/java/classes;" org.testng.TestNG testng.xml
Here are the properties that TestNG understands:
3.4.2. System properties
Property
Type
Documentation
testng.test.classpath
A semicolon separated series of directories that contain your test classes.
If this property is set, TestNG will use it to look for your test classes instead of the class path. This is convenient if you are using the package tag in your XML file and you have a lot of classes in your classpath, most of them not being test classes.
Example:
java org.testng.TestNG -groups windows,linux -testclass org.test.MyTest
The ant task and
testng.xml
allow you to launch TestNG with more parameters (methods to include, specifying parameters, etc…​), so you should consider using the command line only when you are trying to learn about TestNG and you want to get up and running quickly.
The command line flags that specify what tests should be run will be ignored if you also specify a testng.xml file, with the exception of
-groups
and
-excludegroups
, which will override all the group inclusions/exclusions found in
testng.xml
3.4.3. Running tests from within a test jar
TestNG can be provided with a jar that contains your test classes and you can execute the tests from within it.
Let’s see an example (We are going to use Maven for this example.)
We will be using
maven-assembly-plugin
and
maven-jar-plugin
to demonstrate how to do this.
The project’s directory structure will look like below:
├── pom.xml
└── src
└── test
├── java
│ └── org
│ └── testng
│ ├── FirstTestCase.java
│ └── SecondTestCase.java
└── resources
├── suites
│ ├── suite1.xml
│ └── suite2.xml
└── test-jar-with-dependencies.xml
The relevant

section and the

section will look like below:


org.testng
testng
7.9.0
test





org.apache.maven.plugins
maven-jar-plugin
3.3.0



test-jar





org.apache.maven.plugins
maven-assembly-plugin
3.6.0


src/test/resources/test-jar-with-dependencies.xml




org.testng.TestNG





package

single






The contents of
src/test/resources/test-jar-with-dependencies.xml
will look like below:
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/plugins/maven-assembly-plugin/assembly/1.1.0 http://maven.apache.org/xsd/assembly-1.1.0.xsd
test-jar-with-dependencies

jar

false


/
true
true
true
test



Now let’s build the jars using the command
mvn clean package
Now in order to run the tests, use the below command:
java -jar target/uber-testjar-demo-1.0-SNAPSHOT-test-jar-with-dependencies.jar -testjar target/uber-testjar-demo-1.0-SNAPSHOT-tests.jar
This command causes TestNG to look for test classes in the jar and it executes all of them.
Here:
target/uber-testjar-demo-1.0-SNAPSHOT-test-jar-with-dependencies.jar
- Represents the uber/fat jar which contains all the dependencies inside it.
target/uber-testjar-demo-1.0-SNAPSHOT-tests.jar
- Contains all the tests that we created.
-testjar
- This argument informs TestNG that it should look for test classes inside the jar and NOT in the current CLASSPATH.
If you would like to execute a specific suite file that exists in the jar, then use the below command:
java -jar target/uber-testjar-demo-1.0-SNAPSHOT-test-jar-with-dependencies.jar -testjar target/uber-testjar-demo-1.0-SNAPSHOT-tests.jar -xmlpathinjar suites/suite2.xml
Here we are specifying the path to the suite file using the command line argument
-xmlpathinjar
For more details, refer to
this stackoverflow post
3.5. Test methods, Test classes and Test groups
3.5.1. Test methods
Test methods are annotated with @Test. Methods annotated with @Test that happen to return a value will be ignored, unless you set allow-return-values to true in your testng.xml:



or



3.5.2. Test groups
TestNG allows you to perform sophisticated groupings of test methods. Not only can you declare that methods belong to groups, but you can also specify groups that contain other groups. Then TestNG can be invoked and asked to include a certain set of groups (or regular expressions) while excluding another set. This gives you maximum flexibility in how you partition your tests and doesn't require you to recompile anything if you want to run two different sets of tests back to back.
Groups are specified in your
testng.xml
file and can be found either under the

or

tag. Groups specified in the

tag apply to all the

tags underneath. Note that groups are accumulative in these tags: if you specify group "a" in

and "b" in

, then both "a" and "b" will be included.
For example, it is quite common to have at least two categories of tests:
Check-in tests.
These tests should be run before you submit new code. They should typically be fast and just make sure no basic functionality was broken.
Functional tests.
These tests should cover all the functionalities of your software and be run at least once a day, although ideally you would want to run them continuously.
Typically, check-in tests are a subset of functional tests. TestNG allows you to specify this in a very intuitive way with test groups. For example, you could structure your test by saying that your entire test class belongs to the "functest" group, and additionally that a couple of methods belong to the group "checkintest":
public class Test1 {
@Test(groups = { "functest", "checkintest" })
public void testMethod1() {

@Test(groups = {"functest", "checkintest"} )
public void testMethod2() {

@Test(groups = { "functest" })
public void testMethod3() {
Invoking TestNG with










will run all the test methods in that classes, while invoking it with
checkintest
will only run
testMethod1()
and
testMethod2()
Here is another example, using regular expressions this time. Assume that some of your test methods should not be run on Linux, your test would look like:
@Test
public class Test1 {
@Test(groups = { "windows.checkintest" })
public void testWindowsOnly() {

@Test(groups = {"linux.checkintest"} )
public void testLinuxOnly() {

@Test(groups = { "windows.functest" } )
public void testWindowsToo() {
You could use the following
testng.xml
to launch only the Windows methods:

Note: TestNG uses
regular expressions
, and not
wildmats
. Be aware of the difference (for example,
"anything"
is matched by
".
— dot star — and not
).
Method groups
You can also exclude or include individual methods:










This can come in handy to deactivate a single method without having to recompile anything, but I don’t recommend using this technique too much since it makes your testing framework likely to break if you start refactoring your Java code (the regular expressions used in the tags might not match your methods anymore).
3.5.3. Groups of groups
Groups can also include other groups. These groups are called "MetaGroups". For example, you might want to define a group "all" that includes "checkintest" and "functest". "functest" itself will contain the groups "windows" and "linux" while "checkintest will only contain "windows". Here is how you would define this in your property file:

3.5.4. Exclusion groups
TestNG allows you to include groups as well as exclude them.
For example, it is quite usual to have tests that temporarily break because of a recent change, and you don’t have time to fix the breakage yet. 4 However, you do want to have clean runs of your functional tests, so you need to deactivate these tests but keep in mind they will need to be reactivated.
A simple way to solve this problem is to create a group called "broken" and make these test methods belong to it. For example, in the above example, I know that
testMethod2()
is now broken so I want to disable it:
@Test(groups = {"checkintest", "broken"} )
public void testMethod2() {
All I need to do now is to exclude this group from the run:

This way, I will get a clean test run while keeping track of what tests are broken and need to be fixed later.
you can also disable tests on an individual basis by using the "enabled" property available on both
@Test
and
@Before
@After
annotations.
3.5.5. Partial groups
You can define groups at the class level and then add groups at the method level:
@Test(groups = { "checkin-test" })
public class All {

@Test(groups = { "func-test" })
public void method1() {
//test code goes here

public void method2() {
//more code goes here
In this class,
method2()
is part of the group "checkin-test", which is defined at the class level, while
method1()
belongs to both "checkin-test" and "func-test".
3.6. Parameters
TestNG allows you to pass an arbitrary number of parameters to each of your tests using the `@Parameters` annotation.
There are three ways to set these parameters
The testng.xml file
Programmatically
Java system properties
3.6.1. Parameters from testng.xml
If you are using simple values for your parameters, you can specify them in your
testng.xml
@Parameters({ "first-name" })
@Test
public void testSingleString(String firstName) {
System.out.println("Invoked testString " + firstName);
assert "Cedric".equals(firstName);
In this code, we specify that the parameter
firstName
of your Java method should receive the value of the XML parameter called
first-name
. This XML parameter is defined in
testng.xml






The same technique can be used for
@Before
@After
and
@Factory
annotations:
@Parameters({ "datasource", "jdbc-driver" })
@BeforeMethod
public void beforeTest(String ds, String driver) {
m_dataSource = buildDataSource(); // look up the value of datasource
m_jdbcDriver = driver;
This time, the two Java parameter
ds
and
driver
will receive the value given to the properties
datasource
and
jdbc-driver
respectively.
Parameters can be declared optional with the
org.testng.annotations.Optional
annotation:
@Parameters("db")
@Test
public void testNonExistentParameter(@Optional("mysql") String db) {
//more code
If no parameter named "db" is found in your
testng.xml
file, your test method will receive the default value specified inside the
@Optional
annotation: "mysql".
The
@Parameters
annotation can be placed at the following locations:
On any method that already has a
@Test
@Before
@After
or
@Factory
annotation.
On at most one constructor of your test class. In this case, TestNG will invoke this particular constructor with the parameters initialized to the values specified in
testng.xml
whenever it needs to instantiate your test class. This feature can be used to initialize fields inside your classes to values that will then be used by your test methods.
Important:
The XML parameters are mapped to the Java parameters in the same order as they are found in the annotation, and TestNG will issue an error if the numbers don’t match.
Parameters are scoped. In testng.xml, you can declare them either under:

tag or

tag or

tag or

tag.
The order of precedence (lowest to highest) in terms of resolving values for parameters with same names is
--> --> -->
If two parameters have the same name, it’s the one defined in

that has precedence. This is convenient if you need to specify a parameter applicable to all your tests and override its value only for certain test method.
3.6.2. Parameters with DataProviders
Specifying parameters in testng.xml might not be sufficient if you need to pass complex parameters, or parameters that need to be created from Java (complex objects, objects read from a property file or a database, etc…​). In this case, you can use a Data Provider to supply the values you need to test. A Data Provider is a method on your class that returns an array of array of objects. This method is annotated with
@DataProvider
//This method will provide data to any test method that declares that its Data Provider
//is named "test1"
@DataProvider(name = "test1")
public Object[][] createData1() {
return new Object[][] {
{ "Cedric", 36 },
{ "Anne", 37},
};

//This test method declares that its data should be supplied by the Data Provider
//named "test1"
@Test(dataProvider = "test1")
public void verifyData1(String n1, Integer n2) {
System.out.println(n1 + " " + n2);
will print
Cedric 36
Anne 37
@Test
method specifies its Data Provider with the
dataProvider
attribute. This name must correspond to a method on the same class annotated with
@DataProvider(name="…​")
with a matching name.
By default, the data provider will be looked for in the current test class or one of its base classes. If you want to put your data provider in a different class, it needs to be a static method or a class with a non-arg constructor, and you specify the class where it can be found in the
dataProviderClass
attribute:
public class StaticProvider {
@DataProvider(name = "create")
public static Object[][] createData() {
return new Object[][] {
new Object[] { 42 }
};

public class MyTest {
@Test(dataProvider = "create", dataProviderClass = StaticProvider.class)
public void test(Integer n) {
// ...
The data provider supports injection too. TestNG will use the test context for the injection. The Data Provider method can return one of the following types:
An array of array of objects (
Object[][]
) where the first dimension’s size is the number of times the test method will be invoked and the second dimension size contains an array of objects that must be compatible with the parameter types of the test method. This is the case illustrated by the example above.
An
Iterator
. The only difference with
Object[][]
is that an
Iterator
lets you create your test data lazily. TestNG will invoke the iterator and then the test method with the parameters returned by this iterator one by one. This is particularly useful if you have a lot of parameter sets to pass to the method and you don’t want to create all of them upfront.
An array of objects (
Object[]
). This is similar to
Iterator
but causes the test method to be invoked once for each element of the source array.
An
Iterator