C/C++ Test Framework - Google Test & Google Mock

Posted on 2017-09-12 Edited on 2017-09-18

~~Edit~~

Google Test，Google Mock以下简称gtest，gmock。
在接触gtest，gmock之前，测试C/C++ code使用UnitTest++。这是一个很简洁的框架，上手很快。参看另一篇博文UnitTest++简介。测试相关的功能够用，但是没有mock库。这带来的问题是：

测试遗留代码的时候，需要自行fake相关代码。这个在遗留系统很庞大时，要颇费心力。而且过多涉及细节，导致测试极不稳定，系统代码任意的演进，都会导致大堆的测试失败，甚至测试无法进行。
测试case之间无法很好的解耦。结果同样是测试不够稳定。术语是测试代码很“脆弱”。

Java，Python，JavaScript都有自己的mock库。Python的Mock类，Java的Mockito/PowerMock，JavaScript的Sinon。于是，在网上搜索了一下C/C++的Mock库，于是看到了gtest，gmock。然后就有了这一篇。

简介

不用去网上费心找教程，两个项目的文档都非常棒。入口统一在gtest GitHub项目主页上。而且该文档不仅很好的介绍了gtest，gmock的用法，其中还涉及了很多TDD或者UnitTest的真知灼见，很值得读一读。
要使用gtest非常简单：编译出gtest，gmock，再链入你的测试程序。

编译gtest/gmock

gtest，gmock均用cmake来管理跨平台，先用cmake来生成Makefile。用命令cmake -G "Unix Makefiles" /path/to/CMakeList.txt

Makefile

然后按照下面编写Makefile。注意gmock_main是一个main函数来调用所有的test case，省得自己写main函数了。

CC = gcc
CPP = g++
LINK = g++
CFLAGS = -g -Wall -Werror -Wextra -std=gnu99
CPPFLAGS = -g -Wall -Werror -Wextra
LIBS = -L./lib -lgtest -lgmock -lgmock_main -lpthread
C__SOURCES = $(wildcard *.c)
CPPSOURCES = $(wildcard *.cpp)
OBJECTS = $(patsubst %.c, %.o, $(C__SOURCES)) $(patsubst %.cpp, %.o, $(CPPSOURCES))
TARGET = test_exe
first: all
%.o: %.c
    $(CC) $(INCLUDES) -c $(CFLAGS) -o $@ $<
%.o: %.cpp
    $(CPP) $(INCLUDES) -c $(CPPFLAGS) -o $@ $<
all: $(TARGET)
$(TARGET): $(OBJECTS)
    $(LINK) $(CPPFLAGS) $(LIBS) -o $(TARGET) $(OBJECTS)
.PHONY : clean
clean:
    rm -f $(TARGET) $(OBJECTS)

Terms

Meaning	Google Test Term	ISTQB Term
Exercise a particular program path with specific input values and verify the results	TEST()	Test Case
A set of several tests related to one component	Test Case	Test Suite

Test

#include "gtest/gtest.h"
#include "gmock/gmock.h"
using ::testing::Return;
using ::testing::Test;
using ::testing::_;
using ::testing::AtLeast;
TEST(TestCaseName, should_this_test_do)
{
    ...
    EXPECT_STREQ("{}", str);
}

中间的那堆namespace都是gtest/gmock库里定义的matcher宏或者各种有用的宏。

Test Fixture

在测试有重复的时候，就要用到Test Fixture了，也就是setUp / tearDown。

class QueueTest : public ::testing::Test {
 protected:
  virtual void SetUp() {
    q1_.Enqueue(1);
    q2_.Enqueue(2);
    q2_.Enqueue(3);
  }
  // virtual void TearDown() {}
  Queue<int> q0_;
  Queue<int> q1_;
  Queue<int> q2_;
};
TEST_F(QueueTest, IsEmptyInitially) {
  EXPECT_EQ(0, q0_.size());
}
TEST_F(QueueTest, DequeueWorks) {
  int* n = q0_.Dequeue();
  EXPECT_EQ(NULL, n);
  n = q1_.Dequeue();
  ASSERT_TRUE(n != NULL);
  EXPECT_EQ(1, *n);
  EXPECT_EQ(0, q1_.size());
  delete n;
  n = q2_.Dequeue();
  ASSERT_TRUE(n != NULL);
  EXPECT_EQ(2, *n);
  EXPECT_EQ(1, q2_.size());
  delete n;
}

constructor/destructor vs. SetUp/TearDown

When you need to write per-test set-up and tear-down logic, you have the choice between using the test fixture constructor/destructor or SetUp()/TearDown(). The former is usually preferred, as it has the following benefits:
By initializing a member variable in the constructor, we have the option to make it const, which helps prevent accidental changes to its value and makes the tests more obviously correct.
In case we need to subclass the test fixture class, the subclass’ constructor is guaranteed to call the base class’ constructor first, and the subclass’ destructor is guaranteed to call the base class’ destructor afterward. With SetUp()/TearDown(), a subclass may make the mistake of forgetting to call the base class’ SetUp()/TearDown() or call them at the wrong moment.
Benefit for using SetUp/TearDown:
If the tear-down operation could throw an exception, you must use TearDown() as opposed to the destructor, as throwing in a destructor leads to undefined behavior and usually will kill your program right away. Note that many standard libraries (like STL) may throw when exceptions are enabled in the compiler. Therefore you should prefer TearDown() if you want to write portable tests that work with or without exceptions.
The assertion macros throw an exception when flag –gtest_throw_on_failure is specified. Therefore, you shouldn’t use Google Test assertions in a destructor if you plan to run your tests with this flag.
In a constructor or destructor, you cannot make a virtual function call on this object. (You can call a method declared as virtual, but it will be statically bound.) Therefore, if you need to call a method that will be overriden in a derived class, you have to use SetUp()/TearDown().

简言之，在逻辑上，这两组的作用相同，都是每个测试之前之后会做一些处理工作。Constructor/Destructor的好处是提供了继承。setUp/tearDown的好处是可以处理exception，这是不能放在析构函数里的。

SetUpTestCase / TearDownTestCase

Test Case级别的SetUp/TearDown

class FooTest : public ::testing::Test {
 protected:
  // Per-test-case set-up.
  // Called before the first test in this test case.
  // Can be omitted if not needed.
  static void SetUpTestCase() {
    shared_resource_ = new ...;
  }
  // Per-test-case tear-down.
  // Called after the last test in this test case.
  // Can be omitted if not needed.
  static void TearDownTestCase() {
    delete shared_resource_;
    shared_resource_ = NULL;
  }
  // You can define per-test set-up and tear-down logic as usual.
  virtual void SetUp() { ... }
  virtual void TearDown() { ... }
  // Some expensive resource shared by all tests.
  static T* shared_resource_;
};

SetUp/TearDown Environment

First, you subclass the ::testing::Environment class to define a test environment, which knows how to set-up and tear-down:
Then, you register an instance of your environment class with Google Test by calling the ::testing::AddGlobalTestEnvironment() function:
Now, when RUN_ALL_TESTS() is called, it first calls the SetUp() method of the environment object, then runs the tests if there was no fatal failures, and finally calls TearDown() of the environment object.
It’s OK to register multiple environment objects. In this case, their SetUp() will be called in the order they are registered, and their TearDown() will be called in the reverse order.
Note that Google Test takes ownership of the registered environment objects. Therefore do not delete them by yourself.

class Environment {
 public:
  virtual ~Environment() {}
  // Override this to define how to set up the environment.
  virtual void SetUp() {}
  // Override this to define how to tear down the environment.
  virtual void TearDown() {}
};
Environment* AddGlobalTestEnvironment(Environment* env);

断言

有两种断言EXPECT_xxx和ASSERT_xxx。前者会让测试终止，后者不会，只会让测试fail。

gmock

之所以要切到gtest，唯一的原因就是gmock，所以要专开一章重点介绍一下。所有内容均来自于官方文档。内容深度由浅入深，依次如下：

最后还有参考手册:

Cheat Sheet

简介

Google C++ Mocking Framework (or Google Mock for short) is a library (sometimes we also call it a “framework” to make it sound cool) for creating mock classes and using them. It does to C++ what jMock and EasyMock do to Java.

何为Mock？

Mocks are objects pre-programmed with expectations, which form a specification of the calls they are expected to receive.

相应的还有Fake和Stub

Fake objects have working implementations, but usually take some shortcut (perhaps to make the operations less expensive), which makes them not suitable for production. An in-memory file system would be an example of a fake.

gmock的文档里只提到了Fake，从Martin Fowler的文章Mocks Aren’t Stubs中摘录如下:

Stubs provide canned answers to calls made during the test, usually not responding at all to anything outside what’s programmed in for the test.

三者作用相同，都是模拟系统其他部分的功能，达到代码隔离的效果，方便测试。但是Mock的特点是更OO化，也符合TDD或者BDD的思想——针对一个object设置期待，再对齐verify。

Mocks vs. Stubs - from Martin Fowler

In order to use state verification on the stub, I need to make some extra methods on the stub to help with verification. As a result the stub implements MailService but adds extra test methods.
Mock objects always use behavior verification, a stub can go either way. Meszaros refers to stubs that use behavior verification as a Test Spy. The difference is in how exactly the double runs and verifies and I’ll leave that for you to explore on your own.

Getting Started

Class to Mock

class Turtle {
  ...
  virtual ~Turtle() {}
  virtual void PenUp() = 0;
  virtual void PenDown() = 0;
  virtual void Forward(int distance) = 0;
};

Mock class

#include "gmock/gmock.h"  // Brings in Google Mock.
class MockTurtle : public Turtle {
 public:
  ...
  MOCK_METHOD0(PenUp, void());
  MOCK_METHOD0(PenDown, void());
  MOCK_METHOD1(Forward, void(int distance));
};

针对待Mock的Turtle class要注意的是:

Note that the destructor of Turtle must be virtual, as is the case for all classes you intend to inherit from - otherwise the destructor of the derived class will not be called when you delete an object through a base pointer, and you’ll get corrupted program states like memory leaks.

Use it

#include "path/to/mock-turtle.h"
#include "gmock/gmock.h"
#include "gtest/gtest.h"
using ::testing::AtLeast;                     // #1
TEST(PainterTest, CanDrawSomething) {
  MockTurtle turtle;                          // #2
  EXPECT_CALL(turtle, PenDown())              // #3
      .Times(AtLeast(1));
  Painter painter(&turtle);                   // #4
  EXPECT_TRUE(painter.DrawCircle(0, 0, 10));
}                                             // #5
int main(int argc, char** argv) {
  // The following line must be executed to initialize Google Mock
  // (and Google Test) before running the tests.
  ::testing::InitGoogleMock(&argc, argv);
  return RUN_ALL_TESTS();
}

如果你愿意的话，可以自己写main，如果你想偷懒，记得在Makefile里link gmock_main。
按Google的说法，gmock也可以和其他测试框架兼容，但总感觉挺悬的。

一些有用的工具

Marcher
EXPECT_CALL(turtle, Forward(_));
以任意参数调用Forward
EXPECT_CALL(turtle, Forward(Ge(100)));
以大于100的整数调用Forward

Cardinalities: How Many Times Will It Be Called?

If neither WillOnce() nor WillRepeatedly() is in the EXPECT_CALL(), the inferred cardinality is Times(1).
If there are n WillOnce()’s but no WillRepeatedly(), where n >= 1, the cardinality is Times(n).
If there are n WillOnce()’s and one WillRepeatedly(), where n >= 0, the cardinality is Times(AtLeast(n)).

EXPECT_CALL(mockObj, func())
    .Times(AtLeast(1))
    .WillOnce(Return(123))
    .WillRepeatedly(Return(456))

上面这段代码要求func函数至少运行一次，第一次返回123，之后每次返回456。

Important note: The EXPECT_CALL() statement evaluates the action clause only once, even though the action may be performed many times. Therefore you must be careful about side effects. The following may not do what you want:

int n = 100;
EXPECT_CALL(turtle, GetX())
.Times(4)
.WillRepeatedly(Return(n++));

因为Return是宏，所以只会替换一次，所以不管GetX调用几次，返回都是101，而不是101,102,103,…

All Expectations Are Sticky
所谓的sticky就是EXPECT_CALL总是生效的，除非你显示的将其失效。如下，所有的EXPECT_CALL都会生效，那么最后一个会覆盖前面所有的，也就是GetX总会返回10。

using ::testing::Return;
...
for (int i = n; i > 0; i--) {
  EXPECT_CALL(turtle, GetX())
      .WillOnce(Return(10*i));
}

如果希望他返回，30,20,10，…，应该这么写

using ::testing::Return;
...
for (int i = n; i > 0; i--) {
  EXPECT_CALL(turtle, GetX())
    .WillOnce(Return(10*i))
    .RetiresOnSaturation();
}

RetireOnSaturation就是显示的让其失效。还有一个办法：

using ::testing::InSequence;
using ::testing::Return;
...
{
  InSequence s;
  for (int i = 1; i <= n; i++) {
    EXPECT_CALL(turtle, GetX())
        .WillOnce(Return(10*i))
        .RetiresOnSaturation();
  }
}

因为InSequence的关系，在前面的EXPECT_CALL生效以后，就会自动失效，所以不会产生覆盖的效果。
Ordered vs Unordered Calls

using ::testing::InSequence;...
TEST(FooTest, DrawsLineSegment) {
  ...
  {
    InSequence dummy;
    EXPECT_CALL(turtle, PenDown());
    EXPECT_CALL(turtle, Forward(100));
    EXPECT_CALL(turtle, PenUp());
  }
  Foo();
}

Expecting Partially Ordered Calls
Google Mock allows you to impose an arbitrary DAG (directed acyclic graph) on the calls. One way to express the DAG is to
use the After clause of EXPECT_CALL .

using ::testing::Sequence;
Sequence s1, s2;
EXPECT_CALL(foo, A())
    .InSequence(s1, s2);
EXPECT_CALL(bar, B())
    .InSequence(s1);
EXPECT_CALL(bar, C())
    .InSequence(s2);
EXPECT_CALL(foo, D())
    .InSequence(s2);

specifies the following DAG (where s1 is A -> B , and s2 is A -> C -> D ):

     +---> B
     |
A ---|
     |
     +---> C ---> D

Uninteresting Calls
这是gmock报的warning。当针对某一个待测函数设置了EXPECT spec，却并没有调用的时候，就会报这个warning。此时gmock认为你对该函数并没有兴趣，所以就不需要这个EXPECT spec。当然你可以选择忽略这个warning，但我认为出这个warning的时候，多半是test漏写了什么。

Returning Live Values from Mock Methods

using testing::ByRef;
using testing::Return;
class MockFoo : public Foo {
    public:
        MOCK_METHOD0(GetValue, int());
};
...
int x = 0;
MockFoo foo;
EXPECT_CALL(foo, GetValue())
    // .WillRepeatedly(Return(ByRef(x))); X error
    .WillRepeatedly(ReturnPointee(x));
x = 42;
EXPECT_EQ(42, foo.GetValue());

一些测试case

写Unit Test并不像想像的那么简单，并不是调用了框架，针对每个函数写test case就可以。按我目前的理解有以下几种挑战：

Test Case如何解耦。不要有重复测试(overlap)。
例如：在写A函数的时候，写了测试testA，B函数会调用A函数，那么在写完A之后写B的测试testB时，是否要将A mock/fake/stub掉? 如果不将A函数Fake掉，则testA和testB之间就是有overlap。我认为这之间可以有取舍，最佳状态应当是此时，将testA删除，只保留testB。但仍应根据具体情况而定。
如何针对依赖关系进行mock化。
例如出现这样的语句: B = new A，则类B依赖于类A。但类A并没有必要编译进test。因为一旦加入A，则势必会引入更多依赖关系，而导致test编译崩溃。依赖关系的解决无穷无尽。在做Android的单元测试时，可以用PowerMock取代Mockito来Mock构造函数，将构造函数Fake化成类似工厂函数，返回类实例。具体参看这篇博文Android单元测试。
但实际上，按照现在的理解，其实Mock构造函数是不可取的，首先造成被测代码spec不清晰，试想一个构造函数怎么会返回另一个类的实例。其次，在C++中很难做到Mock构造函数。好的做法应当是运用Dependency Injection。例如：

class A
{}
class B
{
    void func()
    {
        A* a = new A;
    }
}

类B应当改写为：

class B
{
    void func(A*)
    {
        ...
    }
}

将类A指针传入，解决dependency的问题。

如何能够让测试稳定，在任意环境下均返回同样的测试结果。
这个一般涉及测试环境的影响。例如调用网络相关的功能，在没有网络的环境就没法进行。再例如测试时操作真实具体文件，则该文件被测试外人为或代码修改，则测试可能就会莫名失败。
针对这些情况，我们应当在测试中尽量避免。例如前者，我们应当对网络接口进行Mock化，后者应当在测试的setUp和tearDown中生成虚假文件用于测试，并在测试完成时做清理。
会因为很小的被测代码改动，而导致大面积测试失败，甚至测试崩溃。
这个就是gmock文档中提到的要针对接口编程，针对接口测试。Robert C·Martin在《敏捷软件开发－原则、模式与实践》一书中有提出：所有的代码都应依赖于抽象接口。因为抽象接口是经过抽象的，相对具体的实现代码较为稳定。而被依赖的代码应该尽可能保持稳定，这样基于之上的代码才不会因为依赖的改动而改动。

下面列出几种我在实际写test case时遇到的情况，在gmock中的解决方案。

按照函数参数返回结果 - Fake

例如:

class A
{
    virtual int func(int a, int b);
}
EXPECT_CALL(mockA, func())
    .WillRepeatedly(a+b);

gmock中可以这样做：Using Functions/Methods/Functors as Actions

using ::testing::_;
using ::testing::Invoke;
class MockFoo : public Foo {
public:
    MOCK_METHOD2(Sum, int(int x, int y));
    MOCK_METHOD1(ComplexJob, bool(int x));
};
int CalculateSum(int x, int y) { return x + y; }
class Helper {
public:
    bool ComplexJob(int x);
};
...
MockFoo foo;
Helper helper;
EXPECT_CALL(foo, Sum(_, _))
    .WillOnce(Invoke(CalculateSum));
EXPECT_CALL(foo, ComplexJob(_))
    .WillOnce(Invoke(&helper, &Helper::ComplexJob));
foo.Sum(5, 6); // Invokes CalculateSum(5, 6).
foo.ComplexJob(10); // Invokes helper.ComplexJob(10);

Mock non-virtual函数

// A simple packet stream class. None of its members is virtual.
class ConcretePacketStream {
    public:
        void AppendPacket(Packet* new_packet);
        const Packet* GetPacket(size_t packet_number) const;
        size_t NumberOfPackets() const;
        ...
};
// A mock packet stream class. It inherits from no other, but defines
// GetPacket() and NumberOfPackets().
class MockPacketStream {
    public:
        MOCK_CONST_METHOD1(GetPacket, const Packet*(size_t packet_number));
        MOCK_CONST_METHOD0(NumberOfPackets, size_t());
        ...
}
template <class PacketStream>
void CreateConnection(PacketStream* stream) { ... }
template <class PacketStream>
class PacketReader {
    public:
    void ReadPackets(PacketStream* stream, size_t packet_num);
};
MockPacketStream mock_stream;
EXPECT_CALL(mock_stream, ...)...;
.. set more expectations on mock_stream ...
PacketReader<MockPacketStream> reader(&mock_stream);
... exercise reader ...

为什么要这么做？
因为只能这么做。普通的mock，要通过继承被测试类，并重写virtual函数来实现。而上面的ConcretePacketStream和MockPacketStream并任何没有关系，也就是说，如果传入后者的指针，不用reinterpret_cast是不能转成前者的指针的。
所以想一个变通的办法，用模板类来定义被测代码，在测试时传入mock类，在生产时，传入真实类。

Mocking Side Effects

EXPECT_CALL(mutator, MutateInt(_))
    .WillOnce(DoAll(SetArgPointee<0>(5), Return(true)));
EXPECT_CALL(mutator, Mutate(NotNull(), 5))
    .WillOnce(SetArrayArgument<0>(values, values + 5));

第一个将MutateInt第一个参数指针指向的int，设为5，并返回true。
第二个将values数组的[0,5)拷贝到参数1指向的地址。
如果仍需要返回，则用DoAll，如下：

EXPECT_CALL(mutator, MutateInt(_))
      .WillOnce(DoAll(SetArgPointee<0>(5),
                      Return(true)));

Selecting an Action’s Arguments

using ::testing::_;
using ::testing::Invoke;
bool MyIsVisibleInQuadrant1(bool visible, const string& name, int x, int y,
const map<pair<int, int>, double>& weight,
double min_weight, double max_wight) {
    return IsVisibleInQuadrant1(visible, x, y);
}.
..
EXPECT_CALL(mock, Foo(_, _, _, _, _, _, _))
    .WillOnce(Invoke(MyIsVisibleInQuadrant1)); // Now it works.

定义自己的adaptor MyIsVisibleInQuadrant1，或者用gmock提供的方法优雅的解决。

using ::testing::_;
using ::testing::Invoke;
using ::testing::WithArgs;
...
EXPECT_CALL(mock, Foo(_, _, _, _, _, _, _))
.WillOnce(WithArgs<0, 2, 3>(Invoke(IsVisibleInQuadrant1)));
// No need to define your own adaptor.

Mocking Private or Protected Methods

class Foo {
    public:
        ...
        virtual bool Transform(Gadget* g) = 0;
    protected:
        virtual void Resume();
    private:
        virtual int GetTimeOut();
};
class MockFoo : public Foo {
    public:
        ...
        MOCK_METHOD1(Transform, bool(Gadget* g));
        // The following must be in the public section, even though the
        // methods are protected or private in the base class.
        MOCK_METHOD0(Resume, void());
        MOCK_METHOD0(GetTimeOut, int());
};

C++ allows a subclass to specify a different access level than the base class on a virtual function.

Misc

Keep in mind that one doesn’t have to verify more than one property in one test. In fact, it’s a good style to verify only one
thing in one test. If you do that, a bug will likely break only one or two tests instead of dozens

When it’s being destroyed, your friendly mock object will automatically verify that all expectations on it have been satisfied,
and will generate Google Test failures if not.

Currently these are only platforms that support the pthreads library (this includes Linux and Mac).

加上命令行参数–gmock_verbose=info可以显示所有EXPECT_CALL的具体调用情况。

Some useful tips in gtest

Selecting Tests

If you set the GTEST_FILTER environment variable or the –gtest_filter flag to a filter string, Google Test will only run the tests whose full names (in the form of TestCaseName.TestName) match the filter.
The format of a filter is a ‘:’-separated list of wildcard patterns (called the positive patterns) optionally followed by a ‘-’ and another ‘:’-separated pattern list (called the negative patterns).

./foo_test Has no flag, and thus runs all its tests.
./foo_test –gtest_filter=* Also runs everything, due to the single match-everything * value.
./foo_test –gtest_filter=FooTest.* Runs everything in test case FooTest.
./foo_test –gtest_filter=Null:Constructor Runs any test whose full name contains either “Null” or “Constructor”.
./foo_test –gtest_filter=-DeathTest. Runs all non-death tests.
./foo_test –gtest_filter=FooTest.*-FooTest.Bar Runs everything in test case FooTest except FooTest.Bar

Temporarily Disabling Tests

// Tests that Foo does Abc.
TEST(FooTest, DISABLED_DoesAbc) { ... }
class DISABLED_BarTest : public ::testing::Test { ... };
// Tests that Bar does Xyz.
TEST_F(DISABLED_BarTest, DoesXyz) { ... }

Temporarily Enabling Disabled Tests

just invoke the test program with the –gtest_also_run_disabled_tests flag or set the GTEST_ALSO_RUN_DISABLED_TESTS environment variable to a value other than 0.

Repeating the Tests


$ foo_test –gtest_repeat=1000	Repeat foo_test 1000 times and don’t stop at failures.
$ foo_test –gtest_repeat=-1	A negative count means repeating forever.
$ foo_test –gtest_repeat=1000 –gtest_break_on_failure	Repeat foo_test 1000 times, stopping at the first failure. This is especially useful when running under a debugger: when the testfails, it will drop into the debugger and you can then inspect variables and stacks.
$ foo_test –gtest_repeat=1000 –gtest_filter=FooBar	Repeat the tests whose name matches the filter 1000 times.