Don't just automate. Orchestrate.

Build a library of humanized functional test scripts that seamlessly mix automated and manual tasks.

start chrome
load page at http://toffee.ksmpartners.com
click link "Features"

Command

Control your browsers with simple, easy-to-understand commands. No programming required.

Compose

Compose your commands into plain text script files. Run them one file at a time, or all at once.

Capture

We don't like writing test reports or trace matrices either. Toffee generates these, complete with screen shot evidence capture.

Collaborate

Organize your scripts into campaigns and invite team members and stakeholders to participate. Track requirements coverage and completion metrics in real time.

Toffee Overview

 

Toffee is a platform for building, executing, and reporting functional tests. It automates testing of HTML web applications using plain-English commands, while allowing the test author to fall back to manual test steps to test things that automation can't easily reach.

A Fresh Approach

Toffee's authors have been automating functional tests for 15 years. We've learned that a narrow focus on automation overlooks the human elements of functional testing, from the testers who step in to do what automation cannot, to the quality auditors who must verify that the test program does what the programmer claims it does. In building Toffee, we set out to orchestrate functional testing across humans and across machines. These are our design goals:

  • Embrace automation. Toffee's humanized scripting language facilitates automation without programming, and bridges the semantic distance between test specification and and implementation. If you can write manual test scripts, you can automate them with Toffee. If you can read manual test scripts, you can understand what a Toffee script is up to.
  • Accept its limits. How can you confirm a screen "looks right" to a human? What if your tests require you to leave the browser to use native applications, or the command line? For those corner cases that automation can't (easily) reach, Toffee supports human-assisted test steps that are audited and recorded alongside automated steps.
  • Go standalone. Good functional tests don't require complex setup of test data, and they need not be run in a specific order. Toffee's variables and random value generators make it easy to make each test stand on its own, so that they can be run in parallel.
  • Show your work. Toffee takes the tedium out of coverage tracking, evidence capture, and test reporting. Generate XML, HTML, and PDF test reports automatically - including screen shots that you take on-demand, or implicitly with every test step.

Applications

We designed Toffee with several flavors of functional tests in mind:

  • System tests, to verify that the system meets all of the requirements in its own functional specification. Toffee's requirements upload mechanism tracks which requirements are tested in which scripts, so that you can generate coverage reports and trace matrices.
  • Acceptance tests, to verify that a system meets all of the customer's business requirements. Acceptance testing is generally used for externally-sourced packaged or bespoke software.
  • Regression tests, to verify that new features or bug fixes don't break existing functionality. Regression tests usually require a high level of automation, and frequently exercise design elements not specified in detail in the functional specification.
 

A Humanized Command Language

 

At the heart of Toffee is a simple yet powerful textual command language for controlling your browser and your test environment. We designed it to be accessible to everyone regardless of their technical proficiency, so we intentionally left out conditionals, loops, and other control structures.

Click any of the links below to see some examples of Toffee commands. The explanations are almost superfluous; that is the point.

Command What it does
start chrome Launch a new Google Chrome browser session with a default name (Session 1, Session 2, etc.), and make it the active session
test that browser version is greater than 41.0 Verifies that the target browser version (Chrome, in this case) is acceptable for the test. (Toffee supports "less than" and "equal to" as well.)
start firefox as "admin" Launch a new Firefox browser session with a user-defined name ("admin") and make it the active session
list sessions List all of the active Sessions, and the browser (Firefox, Chrome, etc.) associated with each
switch to session "admin" Make the browser session named "admin" the active one, and thus the target of subsequent commands
load page at http://toffee.ksmpartners.com/ Open the Toffee home page in the current browser session
quit session "admin" Close the browser associated with the session named "admin"
quit all Close all current sessions and all associated browsers
Command(s) What it does
set textbox "Username" to "scaggiano" Sets the value of an input (textarea, password, or text) with an associated label "Username" exists
click link "Features" Clicks a hyperlink with the text "Features"
ctrl-click option "Apple" in select with id "fruits"

ctrl-click option "Banana" in select with id "fruits"		 These two commands select the "Apple" and "Banana" options in a multi-select enabled list, identified by its HTML "id" attribute
click button in column "Delete" of table with id "Employees" where "First Name" = "John" and "Last Name" = "Doe" Clicks a button in a table column, in a row identified by the values of two its other columns.
click element at "//h5[text()='Rating']/following-sibling::fm-rating[1]/ul/li[5]" Clicks an element identified by its XPath location. (We recommend using alias and include commands to make XPath more usable and less brittle.)
Command What it does
test that textbox "Quantity" equals "3" Tests that a textbox with label "Quantity" has value "3"
test that button with id "btnPlaceOrder" is disabled Tests that a button with a specific id attribute is disabled
test that link "Cancun" does not exist Tests that no hyperlink on the page exists with text "Cancun"
test that text in column "Order Total" of table with id "Orders" where "Order ID"="80903" equals "19.95" Uses table coordinates to check the value of a cell in a table
Command What it does
manual: open test.account@mydomain.com email Prefixing the command with manual: indicates a step that requires human assistance. This can be any step that can't be performed by the other Toffee commands, such as running a SQL query or opening a desktop application. The tester is given the option to pass the step without comment, to pass it with comment, or to fail it with comment.
confirm: received email from reservations@mydomain.com about reservation cancellation. Test steps prefixed with confirm: are almost identical those prefixed with manual:, except they're used for verification purposes. The expected result is documented in the test report as everything that follows the confirm: prefix.
Command What it does
capture screenshot Captures a current screenshot on demand, and saves it for documentation purposes.
turn automatic screen shot capture on Enables implicit screenshot capture after the completion of every step.
Command What it does
prompt for newPassword Prompts the user for the value of a variable called ${newPassword}, displays their entry, and asks them to confirm or re-enter it.
prompt for newPassword with "Please enter a new password" without confirmation Prompts the user for the value of a variable called ${newPassword}, using a customized prompt, but skips confirmation.
save value of textbox with id "txtStudyId" as studyId Captures the current value of an element on the interface as a variable, for use in a later command.
set textbox with id "txtStudySearch" to ${studyId} Looks up the value of the studyId variable and uses it in a command
save random alpha string of length 10 as newStudyId Generates a random alphabetic string of length 10 for use in synthesizing unique data. Toffee's randomization commands also support the generation of reusable alphanumeric, masked, and regex-like strings.
alias [filter only ${rating} star ratings] for click element at "//h5[text()='Rating']/following-sibling::fm-rating[1]/ul/li[${rating}]"

filter only 4 star ratings 		The alias command allows you to build new, easy-to-read commands from those that are complex and brittle (such as this XPath example). This allows you to build a library of reusable commands that firewalls changes to the target DOM from your main scripts.

The first command on the left shows the creation of an alias for a complex command that uses a very complicated XPath expression to locate an element. The second shows the usage of that alias later in the script.
 

A Robust Scripting Engine

 

Sequence Toffee commands into script files, and assemble multiple script files into functional tests. Load them into the Toffee interpreter on the fly, or run them all at once in batch mode.

 
Interpreter

All editions of Toffee include a real-time command interpreter. Enter a command, and receive instant feedback of its success or failure: the "code a little, test a little" approach at light speed. We find this instant feedback very helpful for learning the Toffee language.

On-Demand Script Loader

Save tested Toffee commands in plain-text files, and load them one at a time into the interpreter. We find this especially helpful for creating libraries of commands specific to the application under test, using the Toffee alias command. And, because these are plain-text files, you can use your favorite text editor, and they are version control system friendly.

Batch Script Executor

Run one or all of your scripts at once. Toffee stores the results of each run, at your option, in both XML and HTML format. We use this to create record of our test runs, and to track requirement coverage. It is also very useful for running regularly-scheduled smoke tests to ensure that the application is running smoothly.

Robust Test Output

 

Post the results of testing runs to the web, for reporting to your team and to project stakeholders. Include results of both automated and manual test steps side-by-side in a single report.

 
Publish in Multiple Formats

Toffee Console generates its results in linked HTML documents, for consumption by humans, and in XML format, for consumption by downstream processes. With Toffee Enterprise, upload test results to a server that tracks, reports, and graphs the progress of your test campaign across multiple testers and test rounds.

Track Coverage

Whenever the Toffee Console interpreter detects a "pass" command in a script without failures, it logs the ID of the associated requirement in a coverage.xml document. Toffee Enterprise takes this one step further, generating a trace matrix correlating requirements to the scripts that test them to the test results that prove the system performs as expected.

Show Your Work

All editions of Toffee capture full-screen screen shots and include them in the HTML output. The capture screen shot command captures test evidence on an as-needed basis, and the turn automatic screenshot capture on command captures a screen shot after every step.

Editions

 

Toffee is offered in two editions, targeted at different use cases.

 
Toffee Console

Toffee Console is a command-line tool that runs on Windows, Mac, and Linux (or on any platform that supports the Java Runtime Environment version 8. It does pretty much everything that you see above, minus some advanced multi-user features unique Toffee Enterprise. As such, Toffee Console is ideal for testers working alone or in very small teams.

Toffee Enterprise

Toffee Enterprise is a hosted solution that tracks key test metrics in real time, such as requirements coverage, test campaign completion coverage, and overall pass/fail counts. It tracks these across all your testers on all of your teams, and consolidates these metrics in a heads-up dashboard for project sponsors and stakeholders. Toffee Enterprise is ideal for larger development teams that are geographically distributed across cities or continents.