CWPK #46: Creating the cowpoke Package and Unit Tests

Getting Serious about the Code and Deciding to Say Adios

To this point, we have accumulated a growing roster of methods for extracting and building KBpedia, and utilities to support those processes. With this growing maturity from our Cooking with Python and KBpedia series, it is time for us to put in place a formal testing regime for cowpoke and to take the steps to register it as a formal Python package.

The idea of unit testing is to assemble simple tests of single code functions that may be exercised whenever we deem changes in our code base warrants. These simple programs evaluate against a known results set to determine whether the routine still performs as expected. Unit tests are not a blanket approval of a method, but a way to ascertain whether certain key functions perform as expected. Unit testing is viewed by many as the foundation for integrated tests, the combination of which are one of the most important improvements in software development of the past 30 years.

As with so many other areas, there is a diversity of modules available to aid the testing process in Python. The unittest module is a part of Python’s standard library, and is our basis as well. But we will layer on to that a series of modules that will enable us to guide and develop our unit tests directly through the Spyder IDE.

I am most assuredly an amateur programmer. As I’ve stated before, I have never been paid a dime for writing a line of code. (And, now after more than halfway through this series, you can probably see why!) But since there is a widespread view that unit testing is a best practice, from Day One in my plan for this CWPK series I had slotted in one or two installments to learn and implement some unit tests. I began this particular installment with a high expectation, and indeed wrote most of this intro before I sat down to focus on learning and implementing tests. Yet I reached a conclusion quite contrary to my expectations. I’m writing this last sentence here just as I wrap up this investigation, with a slight taste of ashes that reminds me of our various experiences with the somewhat related area of agile programming. For my purposes and personality, there is just too much process, diversion, and paint-by-the-numbers to make unit testing a formal part of my workflow. I think I can see applications in large team development with mission-critical interdependencies, but my major realization is that I am already doing comprehensive, integrated testing. Unit tests are a diversion and a productivity loss, as I presently see them, in the case of knowledge graph roundtripping.

However, that being said, we still have the imperative to package up our CWPK code, which we have named cowpoke, as a standard Python package that we can readily make available through the common channels of GitHub and pip. We conclude this installment with our efforts in these areas, which now means you have complete and unfettered access to all of the code we have prepared to date through these CWPK installments.

Installing the Environment

To enable Spyder as our unittest interface, I began by installing a package extension specific to that task:

  conda install -c spyder-ide spyder-unittest

The unittest operations in Spyder also requires the pytest module, which is already part of my base installation, but we make sure anyway:

  conda install pytest

You will want to set up a folder under your project for ‘tests’ and to write your test files, often multiples, to this directory for the package. As you install, you may be asked to grant some permissions, and here is where you will configure to point to your project.

You should then logout and restart your computer, and return to your project to continue. The system will also install a separate .pytest_cache directory under your project.

I found, like Python packages in general, the install and addition of the testing modules to be smooth and easy. A new pane gets created (upper right by default) in Spyder, and test run options appear under the Spyder Run menu item.

Anatomy of a Unit Test

By definition, a unit test is limited to a single “unit” often used synonymously as a discrete function or algorithm. Ted Kaminsky nicely summarizes the standard guidance as to what constitutes a good unit test:

1. Tests should only test one thing
2. Each test should be independent and self-contained
3. Refactoring should not break tests
4. Try to achieve maximal coverage with tests.

A commitment to unit tests encourages more public methods and greater piecing apart of routines. The general form of a unit test looks like:

  fixtures
  def test_test
      setup
      assert
      test
      teardown

The pytest module uses ‘fixtures’ as a way to set up input templates of state or connectivity needed as inputs to the function. The unit test function is named, by convention, with a test_ prefix that informs the module a test is available. Though your production routines may favor shorter or more cryptic variable and function names, within the unit test environment best practice is to use longer and descriptive labels, since the tests and how they are being reported occur in a separate testing panel removed in both code and space from the subject routine.

Each test goes through an initial setup portion and then concludes with a teardown, where the temporary test structures are released when the test is done. The actual tests are done against assertions that have pre-determined ‘correct’ results, so that the test can evaluate to pass or fail. Multiple assertions may be evaluated in a given unit test, so more than one pass-fail may be returned. Like unit tests across tools and languages, results that pass are often shown in green on the screen, fails in red.

Determining Where Units Tests Are Applicable

I began my unit test efforts in earnest by first assembling an inventory of cowpoke‘s defined functions to date:

I then began to lay out my plan of attack on paper. When I research such matters I note sources that seem to have good code examples and I will mark them for later consultation, but my initial investigations are spent more on finding clear coding approaches and constructs and generalities or patterns for how to set up things. One of the first observations is that all of my roundtripping routines involved quite a bit of I/O and configuration. I was therefore looking especially for guidance around the idea of ‘fixtures‘ or ‘parameters’ with pytest. A second observation is that most of my utils.py routines are used infrequently, sometimes no more frequently than once every build or three. These were not heavily used routines.

Most of the unit test examples I came across were toy cases, such as adding or multiplying a couple of numbers or concatenating some strings. I tried to focus my investigations on use of CSV files, since that is such a central construct in our knowledge graph approach. I started to see hints that perhaps unit tests are not a good idea for file and I/O purposes. A quote from the user Dunes on StackOverflow seemed to best capture the sense I was gaining from my research: “Unit tests that access the file system are generally not a good idea. This is because the test should be self contained, by making your test data external to the test it’s no longer immediately obvious which test the csv file belongs to or even if it’s still in use.”

Hmmm. I could see that, good idea or not, what I was going to have to do to set up my tests and get them “mocked” up for all of the I/O and data staging I would need was not a trivial matter. It was also perhaps the case that my general roundtripping routines, with their many steps and loops, were already too complex for unit testing. It was beginning to dawn on me that to design my unit tests properly, I would need to further piece apart my existing routines into more atomic functions. Wow, I really did not like that idea, since it would kick me all of the way back to Square One and force me to re-factor all of my code to date. And I had been making such great progress!

I could see that unit testing was not going to be some minor ‘adder’ to improve best practices, but more akin to a whole change in philosophy and approach. It was at minimum looking that I would need to double the size of my code base, learn a bunch of whole new stuff needed by the test machinery, change my design and architecture, and for examples of isolated functions that told me nothing about application-wide behavior and seemed to only test what I already knew to be true. Ouch! This unit test stuff was not looking to be a good deal.

Calling Time Out and Testing Premises

We had similar realizations about the use of agile development in the past. While we are a boutique development shop that tends to work on smaller, bespoke projects, we have also been subcontractors on much larger teams with enterprise-scale budgets and project management. It is sometimes exciting, often lucrative, and too frequently exasperating to work on big, multi-team projects. We understand the discipline needed for larger-scale projects and can see the merit (if lightly applied) of agile approaches. But too often agile is just another way to kill innovation and productivity through too many meetings and process.

I had taken as a given that unit testing was an unalloyed good. But, here I was, barely hours into a concerted investigation, and I was seeing serious red flags. Because I had initially not questioned the premise, I had not specifically looked into criticisms or critics of unit testing. The truth is, I had just taken it all as a given and had not inspected my testing assumptions. I believe in my bones in the merit of tested and vetted information products, but perhaps unit testing was not a way to go in our circumstance. What was indeed best and true here?

So, I shifted my investigations from ‘how to do’ to ‘whether to do’ and discovered more criticism and naysayers than I had imagined. Some of this criticism was now a dozen or more years old. Some of the criticism is empirical, some philosophical or nuanced.

There is apparently a steep learning curve to master unit testing and making it an integral part of the development process. My initial investigations had flagged that prospect in spades. Unit testing sets up its own incentive objective, which can be a good thing, but if not done with the right balance or awareness, can result in mindless code proliferation or developing to the incentive. More public and smaller methods result, that are hard to maintain over time:

Integrated testing can also be made more difficult due to the code fragmentation.

Respected innovators like Donald Knuth have called unit testing “a waste of time.” Past enthusiasts like David Heinemeier Hansson, the developer of Ruby on Rails, now argue that integrated testing is the proper focus. Kaminski, noted above, has also been critical. There have been many others critical of the approach.

A couple of articles by James Coplien on Why Most Unit Testing is a Waste and its seque in 2014 were lightning rods on the topic. There is a more profane approach to the question, but still thoughtful. Even commercial proponents propose additional steps and tools to improve the unit testing experience and results. There appears to be some growing realization that there are boundaries to unit testing and the need for better definitions of where unit tests may be essential or relevant.

Framing Testing in a Different Light

This more open-minded investigation of the question of unit testing has changed my perspective. My impression is that there is a place and likely best practices and methods for doing unit testing. However, an excessive insistence on unit testing may actually be counter-productive by distorting incentives and leading to code proliferation and fragmentation. Paradoxically, this may make the code base harder to maintain and make it more difficult to discover integrated or system issues. One area that concerns me is in RESTful or Web-based distributed development where APIs and interfaces are prominent, but hard to mock up. The lack of examples useful to my needs is another concern.

More fundamentally, this exercise has caused me to think of testing in a new light. I remain convinced that testing and reliability are paramount, but that has meaning only in relation to the ultimate deliverables or purposes, not the individual pieceparts. The objective is the purpose of the software, not unit testing per se.

A roundtripping objective, my governing purpose, is, actually, a system test of the highest order. We need to be able to break down and manipulate a knowledge artifact, re-build it again, and be able to inspect and use it in process-heavy external environments. Being able to load and inspect and apply logic tests in a totally different Protégé environment is a demanding system test for whether our code base has been accurate in the entire cycle of transformations. I’m already doing loads of testing, and relevant, too. My realization was that the entire basis of my CWPK series was to create an artifact, test it for coherence, modify it, and then test it for coherence again. Such roundtripping is indeed a demanding task.

I am glad I began with the premise of instituting some unit tests in the cowpoke project. It has caused me to think more clearly about why test in the first place, and that achieving end goals should take precedence over adhering to any particular method or process. There is no end to the learning, is there?

The conclusion about the immediate objective was to put unit testing off to the side. If I can completely break down and then re-build a knowledge graph, there is no shame in not doing unit testing.

Creating the cowpoke Package

It is not necessary to have a pip package for cowpoke, since it is possible (if you have the GitPython package installed) to obtain the code directly from GitHub:

pip install gitpython

import git
git.Git("/xxx/main-python-directory/Lib/site-packages").clone("git://github.com/Cognonto/cowpoke.git")

However, it is easier to treat cowpoke as a standard Python package, and we created one and did so by following guidelines for the PyPi installer package (pip).

First, I did a test installation at test.pypi.org using this step-by-step guide. There are a few required files that each package must contain, including notably:

setup.py                   # definitions of the package and dependencies
LICENSE                    # the license for the package
README.md                  # the readme description file
code files

All of these requirements and the steps to follow are outlined in the guide.

Windows is a little tricky. I had a hard time using the Apache 2 license, so fell back to the MIT one. Also, the acceptance of tokens, as suggested by the guide, proved problematic, possibly due to lack of a $HOME directory on my Windows machine. I used my straight login and password names for the test site instead, and that worked fine. One must also have the setup.py working just right, or the test will fail with an error. (You can run python setup.py -install to check your pip packages locally.) Also, the instructions kept insisting I use ‘python3‘, but my local configuration sets Python directly to version 3, so the numeral was not causing Python to run properly; using the simple python did the trick for my environment.

Nonetheless, after making these changes, I was able to successfully complete the test install.

This test exercise means the package file structure is now suitable for the actual formal package upload. There is a separate guide for the formal site. Note that the formal package registry has a separate site (https://pypi.org/) with its own login and password than the test site. Per the test site instructions, I had already installed the twine install assistance package. So, after logging into the PyPI support, we begin the upload process with:

python -m twine upload dist/*

I am then prompted for my PyPI login and password. The material is then uploaded with progress bars, and upon acceptance we get a message about where to find our new cowpoke package:

https://pypi.org/project/cowpoke/

Now, it is important to know that one can not update this information without incrementing the version number. So, it is essential that the input information be accurate and complete, which means the test upload is a very important step.

Going forward, it is now possible for you to install cowpoke directly into your Python project by using:

pip install cowpoke

Lastly, please notice I have updated the first notice banner at the conclusion of these installments to indicate where to find the cowpoke Python code.

Additional Documentation

Here are some sources on the general question of testing and unit testing in Python:

• Some guidelines for testing, especially the lead-in
• A tutorial on PyUnit (unittest)
• Understanding Unit Testing
• The unittest plugin for Spyder
• How to run and debug unittest in Spyder.

Here are some sources on how to create a repository on GitHub and create a pip package:

• Packaging Python Projects
• Python Packaging User Guide.

NOTE: This CWPK installment is available both as an online interactive file or as a direct download to use locally. Make sure and pick the correct installment number. For the online interactive option, pick the *.ipynb file. It may take a bit of time for the interactive option to load.

I am at best an amateur with Python. There are likely more efficient methods for coding these steps than what I provide. I encourage you to experiment — which is part of the fun of Python — and to notify me should you make improvements.