Magento2 - My first MFTF Test

Magento2 - My first MFTF Test

June 20, 2020

Introduction #

As Magento developer, you quickly realise that keeping high quality of your work in such complex architecture is both challenging and required at the same time. When your code is buggy, it can affect different aspects of the system. Also, how can you be sure that you are not breaking part that you’re not even aware of?

Magento introduced MFTF framework some time already, and when I wanted to use it, I’ve encountered some issues and support for these issues are somehow limited. I didn’t find many articles or “hello-world” examples on MFTT so I had to figure it out myself. In order not to waste this research, I decided to share some thoughts, guidelines and real-life working example. Hope it helps!

Versioning #

Installation process seemed well descibed in official Magento devdocs, however first obstacle I had was versioning, Magento states:

Find your MFTF version of the MFTF. The latest Magento 2.3.x release supports MFTF 2.5.3. The latest Magento 2.2.x release supports MFTF 2.4.5.

So which version to use for Magento 2.3.2? With 2.5.3 I had many validations errors coming from incompatible tests. I ended up with using Magento 2.3.5 and MFTF 2.6.3 and this combination finally worked flawlessly. However - it raises few questions - MFTF is still young and often changed project and tests need migrations between versions. Last thing you want to have with migration to newer Magento is requirement to update tests.

First test #

I am going to cover one my modules with acceptance test using MFTF. This is small module module that just adds two links in admin notification bar, when cache refresh is needed, you can see how it works below:

Admin cache refresh module in action

So how my test would look like? Well - how human would test it:

  1. Login into admin area
  2. Change something in configuration and save
  3. Wait to see if message about invalidated cache is shown
  4. Click on this link
  5. Go to cache management page and see that there are no invalidated cache types
  6. Logout

Create main testfile #

In order to create test, we need to create directory Test/Mftf - this will be root directory for MFTF based tests. In this directory we need to create another Test directory - it will hold all testcases: Test/Mftf/Test/AdminCacheMessageTest.xml.

At first, we will create very basic test, just to have it working. It will be improved later.

All tests are defined using XML directives, which is really cool, because:

  • tests can be created by people who cannot code - basic XML, CSS, xPath skills are totally enough
  • XSD validation will tell you instantly, when you make mistake
  • tests are short, well described and easy to read

Basic structure for testfile loooks like this:

<?xml version="1.0" encoding="UTF-8"?>
<tests xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xsi:noNamespaceSchemaLocation="urn:magento:mftf:Test/etc/testSchema.xsd">
  <test name="AdminCacheRefreshTest">
      <annotations>
          <features value="Cache"/>
          <stories value="Allows quick cache-refresh in admin panel..."/>
          <title value="Admin should be able to quickly refresh cache without going to cache page..."/>
          <description value="Login to admin, change system config, check if notification contains link to quick cache refresh."/>
          <severity value="MINOR"/>
          <group value="Cache"/>
      </annotations>
      <before>...</before>
      <after>...</after>
  </test>
</tests>

I have filled content of annotations nodes with information about my test - title, description, what will it do and what parts of system it affects. You can link it with tickets and assign severity.

Before and after #

In before and after nodes you defined steps which should take place before or after actual place, something like preparations for the actual test.

Steps defined here will be executed always, even if test fail. So for example, for your feature you might need to create product - which will be used in test - but - no matter if test succeeds or fails - you must remove your product at the end, so when test is executed again - it will have clean base:

Create product ➡️ Run test ➡️ Remove product (over and over again).

In our case - let’s think what needs to happen before and after test - we need to ensure that cache gets validated - we will change some config value - let’s say we set Default country to Germany and save. At this point we know configuration file must be refreshed and this is our starting point for actual test:

<before>
  <actionGroup ref="LoginAsAdmin" stepKey="logInAsAdmin"/>
  <amOnPage url="{{AdminConfigGeneralPage.url}}" stepKey="goToSystemConfig"/>
  <uncheckOption selector="input#general_country_default_inherit" stepKey="uncheckInheritanceCheckbox"/>
  <selectOption userInput="DE" selector="select#general_country_default" stepKey="checkGermanyAsDefaultCountry"/>
  <click selector="button#save" stepKey="SaveConfig" />
</before>

So what is exactly happening here? Let’s go line by line:

<actionGroup ref="LoginAsAdmin" stepKey="logInAsAdmin"/>

ActionGroup directive allows us to use some step of steps defined somewhere else. LoginAsAdmin is one the most used ActionGroups - there is no need that every vendor or developer must recreate same steps required for login as admin? It’s always the same - open backend url, fill out username and password, click on login. These ActionGroups are written for most common tasks by Magento for their modules and they are used by their own respective tests. List of all ActionGroups can be found here, conveniently grouped by module. So if you need to test something in cart, you don’t have to create product, create category, open category, open product, add to cart - just use exising action group, for example - StorefrontAddProductToCartActionGroup and with just one line in XML you’ll have the job done.

stepKey attribute is an unique step identifier - it will be useful later, for now just keep it unique and descriptive (what step actually does).

<amOnPage url="{{AdminConfigGeneralPage.url}}" stepKey="goToSystemConfig"/>

amOnPage directive just redirects browser to specfied url. You might wonder how the url is specified - we will cover it later.

<uncheckOption selector="input#general_country_default_inherit" stepKey="uncheckInheritanceCheckbox"/>

This directive will uncheck inheritance checkbox, please notice - that just by reading this line you know exactly what it’s gonna do - everything is clear.

<selectOption userInput="DE" selector="select#general_country_default" stepKey="checkGermanyAsDefaultCountry"/>

Just like above - it’s self explanatory - we tell webdriver to select option to value DE and we provide CSS selector for this select.

<click selector="button#save" stepKey="SaveConfig" />

Nothing to add 😄

So at this point we have our testcase prepared - but before we start with actual test - let’s prepare rollback script, to rever the change we did in before script.

 <after>
     <amOnPage url="{{AdminConfigGeneralPage.url}}" stepKey="goToSystemConfigAgain"/>
     <checkOption selector="input#general_country_default_inherit" stepKey="checkInheritanceCheckbox"/>
     <click selector="button#save" stepKey="SaveConfigAgain" />
     <actionGroup ref="logout" stepKey="logoutFromAdmin"/>
 </after>

It’s not necessary to explain this line by line as it basically does the same as before script, except - in reverse. At the end we logout from admin.

Main test routine #

As defined before, our actual test could look like this:

<amOnPage url="{{AdminCacheManagementPage.url}}" stepKey="goToSystemConfigAgain"/>
<waitForPageLoad stepKey="waitForPageLoad"/>
<click selector=".message-system-action-dropdown" stepKey="clickOnMessagesToggle"/>
<see userInput="Or - you can click here " selector=".message-system-list" stepKey="seeExpectedText"/>
<seeLink userInput="just invalidated ones" stepKey="seeJustInvalidatedLink"/>
<seeLink userInput="refresh all cache types" stepKey="seeAllLink"/>

<click selectorArray="['link' => 'just invalidated ones']" stepKey="clickRefreshInvalidatedCaches"/>
<waitForAjaxLoad stepKey="waitForAjaxLoad"/>
<dontSee userInput="Or - you can click here " selector=".message-system-list" stepKey="dontSeeExpectedText"/>

<amOnPage url="{{AdminCacheManagementPage.url}}" stepKey="goToCacheConfig"/>
<dontSee userInput="Invalidated" selector="#cache_grid_table" stepKey="dontSeeInvalidatedCache" />

Lots of cool stuff happening here, let’s go line by line again:

<amOnPage url="admin/system_config/" stepKey="goToSystemConfigAgain"/>

In theory after finishing before routine, I should already be on this page, but I believe it’s good practise to entry point defined in the beginning of actual test too.

<waitForPageLoad stepKey="waitForPageLoad"/>

This directive is really important - as you know Magento does lots of stuff in background - JS templates are being loaded, AJAX calls are processed and so on. Sometimes, you must tell webdriver explicitely to wait until all of this is done. Otherwise, if you want to click on element in next step - it might turn out that it is not rendered yet. waitForPageLoad ensures that page is fully loaded before processing next step.

<click selector=".message-system-action-dropdown" stepKey="clickOnMessagesToggle"/>

Since there might be more than one message waiting for current admin, we want to click on dropdown item to display all of them.

<seeLink userInput="just invalidated ones" stepKey="seeJustInvalidatedLink"/>

Finally - our first assertion! Here we check if messages block contain link added by our module - in userInput we provide link’s text. seeLink checks whether defined link is rendered and visible.

<seeLink userInput="refresh all cache types" stepKey="seeAllLink"/> 

Yet another assertion, to make sure we also see the second link added by our module.

<click selectorArray="['link' => 'just invalidated ones']" stepKey="clickRefreshInvalidatedCaches"/>

When we ensured that both links are visible, now it’s time to validate whether they really work as expected - so we click on one of them. Please notice new handy argument - selectorArray.

<waitForAjaxLoad stepKey="waitForAjaxLoad"/>

Similiar to waitForPageLoad - here we ask driver to wait until AJAX call is finished.

<dontSee userInput="Or - you can click here " selector=".message-system-list" stepKey="dontSeeExpectedText"/>

In our module logic, when cache is cleared, we hide notifications layer - and this directive asserts that we no longer can see text rendered by our module.

<amOnPage url="{{AdminCacheManagementPage.url}}" stepKey="goToCacheConfig"/>

Now we go to cache management page…

<dontSee userInput="Invalidated" selector="#cache_grid_table" stepKey="dontSeeInvalidatedCache" />

And here we make sure that in admin grid we don’t see Invalidated keyword - what means that our module actually worked and invalidated cache types got refreshed.

So here we are, this is our testcase implemented entirely in XML. When we execute it with:

vendor/bin/mftf run:test AdminCacheRefreshTest

Chrome (if not other browser defined) will open and all those steps will be executed and if everything goes fine, you should see output like this in your console:

AdminCacheRefreshTestCest: Admin cache refresh test
Signature: Magento\AcceptanceTest\_default\Backend\AdminCacheRefreshTestCest:AdminCacheRefreshTest
Test: tests/functional/Magento/FunctionalTest/_generated/default/AdminCacheRefreshTestCest.php:AdminCacheRefreshTest
Scenario --
[logInAsAdmin] LoginAsAdmin
  [navigateToAdmin] am on page "/csadmin/admin"
  [fillUsername] fill field "#username","admin"
  [fillPassword] fill field "#login","********"
  [clickLogin] click ".actions .action-primary"
  [clickLoginWaitForPageLoad] wait for page load 30
  [clickDontAllowButtonIfVisible] conditional click ".modal-popup .action-secondary",".modal-popup .action-secondary",true
  [closeAdminNotification] close admin notification
[goToSystemConfig] am on page "/csadmin/admin/system_config/"
[uncheckInheritanceCheckbox] uncheck option "input#general_country_default_inherit"
[checkGermanyAsDefaultCountry] select option "select#general_country_default","DE"
[SaveConfig] click "#save"
[goToSystemConfigAgain] am on page "/csadmin/admin/system_config/"
[waitForPageLoad] wait for page load 30
[clickOnMessagesToggle] click ".message-system-action-dropdown"
[seeExpectedText] see "Or - you can click here ",".message-system-list"
[seeJustInvalidatedLink] see link "just invalidated ones"
[seeAllLink] see link "refresh all cache types"
[clickRefreshInvalidatedCaches] click {"link":"just invalidated ones"}
[waitForAjaxLoad] wait for ajax load 30
[dontSeeExpectedText] don't see "Or - you can click here ",".message-system-list"
[goToCacheConfig] am on page "/csadmin/admin/cache"
[dontSeeInvalidatedCache] don't see "Invalidated","#cache_grid_table"
[goToSystemConfigAgain] am on page "/csadmin/admin/system_config/edit/section/general/"
[checkInheritanceCheckbox] check option "input#general_country_default_inherit"
[SaveConfigAgain] click "#save"
[logoutFromAdmin] logout
  [amOnLogoutPage] am on page "/csadmin/admin/auth/logout/"
 PASSED

--------------------------------------------------------------------------------

Time: 2.32 minutes, Memory: 8.00MB

OK (1 test, 5 assertions)

Rendered code doesn’t show it, but you will see lots of green in your console around PASSED keyword, which is the best reward developer can get!

Improvements #

So, we have working test - we could easily end here and move on. Our module is simple, and one test is just enough to test it - however in most cases - for your customization you will have to write more than one test - and because of that there are few things to explain and improve. In this line, we used something like this:

<amOnPage url="{{AdminCacheManagementPage.url}}" stepKey="goToSystemConfig"/>

As good practise, we don’t use hardcoded url - if we would provide direct url for this controller - custombackend/admin/system_config/edit/section/general/ - it would not work. MFTF offers option to define Pages - where we define structure for pages and sections. In our case, we created file Page/AdminConfigurationPage.xml where we defined page for General system config:

<?xml version="1.0" encoding="utf-8"?>
<pages xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:mftf:Page/etc/PageObject.xsd">
    <page name="AdminConfigurationPage" url="admin/system_config/" area="admin" module="Magento_Config">
        <section name="GeneralConfigSection" />
    </page>
</pages>

Of course - it’s also XML. Please notice - that we defined area as admin - in this case MFTF will treat it differently and include custom backend url defined in app/dev/acceptance/.env Using Pages is not obligatory, but definitely recommended:

  • you are able to define area of the page
  • you can defined sections for your pages
  • in your tests you are using references to page’s XML structure - so if you need to change some URL - you’ll have to it once in page xml, and not in all of your test files.
  • you can re-use pages in other tests.

To the last point - one of greatest features of MFTF is re-usability. In my example, I have defined new Page for system configuration - I did it only for purposes of demonstration how Page XMLs work. Recommended way is to reuse existing files. Magento team, when creating tests for module-config, already had to define pages for their own purposes - see vendor/magento/module-config/Test/Mftf/Page directory. You will easily find AdminConfigPage.xml with our page already defined:

<page name="AdminConfigPage" url="admin/system_config/" area="admin" module="Magento_Config">
    <section name="AdminConfigSection"/>
</page>

So if we reference this file and URL - in case Magento for some reason changes URLs in future we won’t have to updates URLs in tests.

The same goes with Sections - MFTF uses Sections to collect used page elements (buttons, links, grids) - all the stuff that is used for interactions should be defined in sections, not in tests directly. For example, our section could look like this:

<?xml version="1.0" encoding="utf-8"?>
<sections xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:mftf:Page/etc/SectionObject.xsd">
    <section name="GeneralConfigSection">
        <element name="saveButton" type="button" selector="#save"/>
        <element name="defaultCountryInheritanceCheckbox" type="input" selector="input#general_country_default_inherit"/>
        <element name="defaultCountrySelect" type="select" selector="select#general_country_default"/>
        <element name="messageDropdownToggle" type="button" selector=".message-system-action-dropdown" />
    </section>
</sections>

As you can see, it contains elements which I need for my test - save button, country select, inheritance checkbox. After all improvements, my testcase finally looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<tests xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:noNamespaceSchemaLocation="urn:magento:mftf:Test/etc/testSchema.xsd">
    <test name="AdminCacheRefreshTest">
        <annotations>
            <features value="Cache"/>
            <stories value="Allows quick cache-refresh in admin panel..."/>
            <title value="Admin should be able to quickly refresh cache without going to cache page..."/>
            <description value="Login to admin, change system config, check if notification contains link to quick cache refresh."/>
            <severity value="MINOR"/>
            <group value="Cache"/>
        </annotations>
        <before>
            <actionGroup ref="LoginAsAdmin" stepKey="logInAsAdmin"/>
            <amOnPage url="{{AdminConfigurationPage.url}}" stepKey="goToSystemConfig"/>
            <uncheckOption selector="{{GeneralConfigSection.defaultCountryInheritanceCheckbox}}" stepKey="uncheckInheritanceCheckbox"/>
            <selectOption userInput="DE" selector="{{GeneralConfigSection.defaultCountrySelect}}" stepKey="checkGermanyAsDefaultCountry"/>
            <click selector="{{GeneralConfigSection.saveButton}}" stepKey="SaveConfig" />
        </before>

        <amOnPage url="{{AdminConfigurationPage.url}}" stepKey="goToSystemConfigAgain"/>
        <waitForPageLoad stepKey="waitForPageLoad"/>
        <click selector="{{GeneralConfigSection.messageDropdownToggle}}" stepKey="clickOnMessagesToggle"/>
        <see userInput="Or - you can click here " selector="{{GeneralConfigSection.messagesSystemList}}" stepKey="seeExpectedText"/>
        <seeLink userInput="just invalidated ones" stepKey="seeJustInvalidatedLink"/>
        <seeLink userInput="refresh all cache types" stepKey="seeAllLink"/>

        <click selectorArray="['link' => 'just invalidated ones']" stepKey="clickRefreshInvalidatedCaches"/>
        <waitForAjaxLoad stepKey="waitForAjaxLoad"/>
        <dontSee userInput="Or - you can click here " selector="{{GeneralConfigSection.messagesSystemList}}" stepKey="dontSeeExpectedText"/>

        <amOnPage url="{{AdminCacheManagementPage.url}}" stepKey="goToCacheConfig"/>
        <dontSee userInput="Invalidated" selector="#cache_grid_table" stepKey="dontSeeInvalidatedCache" />

        <after>
            <amOnPage url="{{AdminConfigGeneralPage.url}}" stepKey="goToSystemConfigAgain"/>
            <checkOption selector="{{GeneralConfigSection.defaultCountryInheritanceCheckbox}}" stepKey="checkInheritanceCheckbox"/>
            <click selector="{{GeneralConfigSection.saveButton}}" stepKey="SaveConfigAgain" />
            <actionGroup ref="logout" stepKey="logoutFromAdmin"/>
        </after>
    </test>
</tests>

As you can see this test doesn’t contain any hardcoded URLs or selectors. What else could be improved? MFTF recommends using ActionGroups - extracting small portions of steps into defined bigger routines - especially those which can possibly be reused. For example - LoginAsAdmin is useful actionGroup. If you have some actions that you will have to repeat - it’s nice to extract it - it will not only make your tests more flexible, but also it will make your tests shorter and easier to read and understand. Let’s put our before and after routines into respective ActionGroups:

Test/Mftf/ActionGroup/SetDifferentCountryAsDefaultActionGroup.xml

<?xml version="1.0" encoding="UTF-8"?>
<actionGroups xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
              xsi:noNamespaceSchemaLocation="urn:magento:mftf:Test/etc/actionGroupSchema.xsd">
    <actionGroup name="SetDifferentCountryAsDefaultActionGroup">
        <annotations>
            <description>Goes to system config page, unchecks Use default value checkbox, sets default country to the one sent as argument.</description>
        </annotations>

        <arguments>
            <argument name="country_id" type="string"/>
        </arguments>

        <amOnPage url="{{AdminConfigurationPage.url}}" stepKey="goToSystemConfig"/>
        <waitForPageLoad stepKey="waitForPageLoad"/>
        <uncheckOption selector="{{GeneralConfigSection.defaultCountryInheritanceCheckbox}}" stepKey="uncheckInheritanceCheckbox"/>
        <selectOption userInput="{{country_id}}" selector="{{GeneralConfigSection.defaultCountrySelect}}" stepKey="checkGermanyAsDefaultCountry"/>
        <click selector="{{GeneralConfigSection.saveButton}}" stepKey="SaveConfig" />
    </actionGroup>
</actionGroups>

💡 Few things to mention here:

  • it is considered as good practise to add ActionGroup suffix to filename and name - to differentiate them between tests and action groups;
  • you can add annotation to briefly describe what is being done here
  • you can parametrize action groups - here - we set country_id argument which we expect to get from testcase

With parameters - you can notice that you can further re-use ActionGroups for other tests which might require different params - in this case you just change argument and you’re done.

We do the same with after routine:

Test/Mftf/ActionGroup/SetDifferentCountryAsDefaultOneActionGroup.xml

<?xml version="1.0" encoding="UTF-8"?>
<actionGroups xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:noNamespaceSchemaLocation="urn:magento:mftf:Test/etc/actionGroupSchema.xsd">
 <actionGroup name="SetDifferentCountryAsDefaultOneActionGroup">
   <annotations>
     <description>Goes to system config page, checks Use default value checkbox, saves config.</description>
   </annotations>

   <amOnPage url="{{AdminConfigurationPage.url}}" stepKey="goToSystemConfig"/>
   <checkOption selector="{{GeneralConfigSection.defaultCountryInheritanceCheckbox}}" stepKey="checkInheritanceCheckbox"/>
   <click selector="{{GeneralConfigSection.saveButton}}" stepKey="SaveConfigAgain" />
 </actionGroup>
</actionGroups>

So, after these improvements, our testcase XML finally looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<tests xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:noNamespaceSchemaLocation="urn:magento:mftf:Test/etc/testSchema.xsd">
    <test name="AdminCacheRefreshTest">
        <annotations>
            <features value="Cache"/>
            <stories value="Allows quick cache-refresh in admin panel..."/>
            <title value="Admin should be able to quickly refresh cache without going to cache page..."/>
            <description value="Login to admin, change system config, check if notification contains link to quick cache refresh."/>
            <severity value="MINOR"/>
            <group value="Cache"/>
        </annotations>
        <before>
            <actionGroup ref="LoginAsAdmin" stepKey="logInAsAdmin"/>
            <actionGroup ref="SetDifferentCountryAsDefaultActionGroup" stepKey="setGermanyAsDefaultCountryActionGroup" >
                <argument name="country_id" value="DE" />
            </actionGroup>
        </before>

        <amOnPage url="{{AdminConfigurationPage.url}}" stepKey="goToSystemConfigAgain"/>
        <waitForPageLoad stepKey="waitForPageLoad"/>
        <click selector="{{GeneralConfigSection.messageDropdownToggle}}" stepKey="clickOnMessagesToggle"/>
        <see userInput="Or - you can click here " selector=".message-system-list" stepKey="seeExpectedText"/>
        <seeLink userInput="just invalidated ones" stepKey="seeJustInvalidatedLink"/>
        <seeLink userInput="refresh all cache types" stepKey="seeAllLink"/>

        <click selectorArray="['link' => 'just invalidated ones']" stepKey="clickRefreshInvalidatedCaches"/>
        <waitForAjaxLoad stepKey="waitForAjaxLoad"/>
        <dontSee userInput="Or - you can click here " selector=".message-system-list" stepKey="dontSeeExpectedText"/>

        <amOnPage url="{{AdminCacheManagementPage.url}}" stepKey="goToCacheConfig"/>
        <dontSee userInput="Invalidated" selector="#cache_grid_table" stepKey="dontSeeInvalidatedCache" />

        <after>
            <actionGroup ref="SetDifferentCountryAsDefaultOneActionGroup" stepKey="restoreDefaultDefaultCountry" />
            <actionGroup ref="logout" stepKey="logoutFromAdmin"/>
        </after>
    </test>
</tests>

If you skip annotations and declarations and focus just one on actual steps, you can see that is really easy to read and understand what is happening. Thanks to extracting before and after routines - test got shorter and cleaner - and in case we want to have more tests - we can reuse a lot! Syntax and vocabulary is easy to read even for non technical people, which I find the best feature of MFTF tests.

Final words #

So, here we are - our test is working nicely - it does what we wanted it to do:

  • 5 XML files
  • 1 test, 5 assertions
  • 1 very long blogpost

It took me about 10h to install, configure, understand basics and implement test I needed - is it much? Not really - MFTF made really good impression and with every next extension it will get easier and faster to create new tests. In terms of efficiency - I only had one test - and MFTF improves efficiency and speed when you need to create more tests - I defined pages, sections and action groups to be used just once (for learning and demonstration purposes) - if I would create more tests - I would reuse those and next steps would be created much faster. I encourage you to give MFTF a shot!

🎬 Click here to see video of performing test (67mb).

🍺 If you liked this article you might consider buying me a beer? ;)