Ontologies within the OBO library are becoming less monolithic and more interlinked. This gives rise to a need for continuous integration tests, in which an ontology is checked for both internal consistency and consistency with related ontologies and information sources. We report here on an implementation of a continuous integration system for OBO libraries using the Jenkins/Hudson web-based system. We present examples of how the system is being used to assist in the development and maintenance of the Gene Ontology and the Human Phenotype Ontology.
Christopher J. Mungall 1*, Heiko Dietze 1, Seth J Carbon 1, Amelia Ireland 1, Sebastian Bauer 2 and Suzanna Lewis 1
1Genomics Division, Lawrence Berkeley National Laboratory, 1 Cyclotron Road, Berkeley, CA 94720 USA.
2Institute for Medical and Human Genetics, Charit ´e-Universit ¨atsmedizin Berlin, Augustenburger Platz 1, 13353 Berlin, Germany.
Many bio-ontologies are transforming from monolithic, standalone artifacts to modular, integrated components in a wider information ecosystem. For example, the Gene Ontology (GO) is making use of orthogonal ontologies such as CHEBI and the Cell Ontology (CL) for automated classification of terms. Previously a GO curator would manually classify xanthophyll biosynthetic process as a subclass of careotenoid biosynthetic process, whereas now this classification can be performed automatically by an OWL reasoner, making use of logical definitions in the GO, and the CHEBI classification of xanthophyll as a subclass of careotenoid.
This represents a major step forward. However, as systems become more interdependent, there is in an increased chance of an error in one component spreading to other components. Consider a hypothetical scenario in which a future version of CHEBI introduces an erroneous classification of xanthine as a subclass of xanthophyll – this error would propagate to the classification of xanthine biosynthetic process in GO. This could have further knock-on effects. GO includes a taxonomic restriction which states that careotenoid biosynthetic process never occurs in Metazoa. This means that a perfectly valid mouse gene annotated to xanthine biosynthesis could be filtered out based on an incorrect classification under careotenoid biosynthetic process (Fig 1). This mouse gene annotation may itself be the basis for further inferences.
Currently many of the inference steps above are not yet fully automated, which provides a buffer against this kind of error propagation. But this also makes it harder to detect the errors further down the line. Ideally we would have some kind of “early warning” system that fails early and allows problems to be detected at the source.
Software engineers use a variety of mechanisms to automate quality control – these include strongly typed languages, which prevent certain kinds of errors at compilation time (and, in modern Integrated Development Environments, during software development), and unit test frameworks such as JUnit1.
Can ontology engineers learn from software engineers? Whilst there are clear differences between ontologies and software, there are also parallels from which we can learn – for example, principles of modularity and normalization. Strong typing in programming can be considered as analogous to rigorous classification using an upper ontology together with constraints in the form of OWL axioms such as domain, range, universal restriction and disjointness axioms. These axioms are used by automated reasoners executed during an ontology editing session, within an ontology development environment such as Protege4.
Fig. 1. Modularization and automated inference. Classifications in GO can be made automatically based on CHEBI (e.g. xanthophyll SubClassOf carotenoid). Figure shows hypothetical scenario in which in an incorrect assertion about “xanthine” is propagated to GO and to gene associations inferences made using GO.
One limitation of this model is that error checking is local to the ontology being developed. In our scenario above, the hypothetical error may be detected by disjointness axioms local to CHEBI, but there is no formal testing of the consequences of assertions on downstream ontologies.
One phase of the software engineering lifecycle is integration testing. During this phase, additional tests are run that make sure the components works when integrated into a larger system – this is analogous to our scenario above. In a traditional waterfall software development model, integration testing is performed at a distinct phase of the lifecycle after development. An alternative is to perform continuous integration (CI) to avoid problems with retrospective integration. In a CI approach, integration tests are woven into the development phase, rather than executed after development. There are a variety of continuous integration environments, for our implementation we opted to go with the Hudson/Jenkins web-based system2. This system has a large number of users in the software engineering community, and comes with a large library of plugins. We did not systematically compare the features with other systems.
Jenkins allows the configuration of multiple jobs (for example, corresponding to software projects, or a set of integration tests for a software project). Each job is associated with a number of builds which are triggered by some event such as a commit to a Version Control System (VCS). Jenkins is integrated with most VCSs, including CVS, Subversion and git. Jobs can also be triggered by other jobs, allowing the configuration of interdependent workflows. For a full list of features, see the Jenkins homepage.
To our knowledge, the concept of continuous integration has never been applied in an ontology development context. Here we report on our efforts to apply the CI environment Jenkins to ontology development – specifically for the GO and the Human Phenotype ontology.
We deployed a standard Jenkins server and created custom jobs for each ontology or information resource3. Jenkins can monitor version control system (VCS) repositories such as the GO subversion repository, so we typically configure each job to execute whenever a change in the relevant files in the repository is detected.
We created a command line wrapper on top of the OWL API as part of the OWLTools framework, and also developed a command line version of the OBO Ontology Release Tool (OORT)4. Jenkins is easily configured to run command line tools.
Figure 2 shows the front page for the Berkeley Jenkins environent.
The front page shows a summary of all jobs – we have organized the jobs into categories such as GO, software, etc. Next to each job are visual indicators stating whether the last build was a success or failure (blue or red ball) plus a weather icon showing “outlook” based on the last five builds (sunny if there have been multiple consecutive successful builds).
Fig. 2. Screenshot for Berkeley Onto-Jenkins environment.
For many of the GO integration tests we make use of GO extensions – these are auxiliary ontologies with axioms that are maintained outside the core ontology5.
The current core version of the GO lacks the kind of strong axiomatization that is useful for detection of problems via automated reasoning. This is partly for historic reasons – OBO-Edit is still widely used, and does not handle disjointness axioms well. We therefore created an extension ontology to the GO called x-disjoint.owl which includes class disjointness axioms such as “nothing is both an organelle and a cell”. This also includes spatial disjointness axioms such as “nothing that is part of a nucleus is part of a cytoplasm”.
These can be encoded as OWL General Class Inclusion (GCI) axioms such as:
(‘part of’ some cytosol)
(‘part of’ some nucleus)
In fact, the existence of some named classes such as “nuclear part” in the GO obviates the need for GCIs, although these named classes may disappear in the future.
Another strong axiomatization source is the GO Taxon constraints. These statements make use of two shortcut relations: never in taxon and only in taxon. Shortcut relations can be expanded by Oort.6
Figure 3 shows the time taken to run HermiT (1.3.4) over the extended GO. This illustrates how some heavy-duty reasoning tasks can be shunted to the server, rather than executed locally on the ontology authors’ machines.
In addition to reasoner checks, the Jenkins environment also takes care of running additional syntactic and structural checks on both ontologies and annotations9. Many of these are implemented as standalone perl scripts. For example, one test ensures that every ID cross-referenced in the GO uses an ID-space registered in the central GO registry of databases10.
In addition to performing validation tests, Jenkins can produce builds, i.e. derived files. We use Oort to make obo and owl versions of the ontology, together with subsets of the ontology.
The GO editorial team are frequent users of the Jenkins system. Whenever a GO editor checks in changes to the ontology it triggers a number of dependent Jenkins jobs. The first job performs basic tests on the new version of the ontology, this is followed by more detailed integration tests described above. If the build fails an email is sent to the editors mailing list. When a corrected version of the ontology is checked in and the build succeeds Jenkins sends a follow-up email indicating that the problem has been corrected.
The Human Phenotype ontology (HPO) models phenotypic features to describe Human genetic diseases. [17, 18] In addition to providing a hierarchical vocabulary that can serve as a starting point for clinical diagnostics applications, the project aims to integrate formal definitions for each term using the mechanism proposed in  in order to enable semantic knowledge integration with phenotypic ontologies for other model organisms and thus with the data associated to them.
We wrap the process of preparing the distribution of the HPO into a standard makefile. This allows us to build, test, and package the ontology locally on the curator’s computer, but also to perform the same steps on a CI environment like Jenkins, which triggers the build as soon as tracked files are changed. Moreover, the practice to define the build process in separate file aids documentation and versioning purposes.
The current build process for the HPO involves simple syntactic checks using OORT and in-house perl scripts as well as semantic checks using the GULO application. GULO compares the subsumption structure inferred from the logical definitions using terms from the building block ontologies with the manually asserted hierarchy in order to quantify the agreement of both knowledge representations.
Fig. 3. Plot of time taken by HermiT for consistency checking on extended GO for builds 457-461. These are generated automatically by the CI system.
The makefile in invoked as part of a freestyle Jenkins project. The CompilerWarnings plugin of Jenkins is configured to parse the output produced during the build and to turn the logged information into a statistics using a simple Groovy script. The statistics is interpreted by Jenkins to assess the general stability of the build, which in turn has effects on the deployment. For instance, if a severe error is detected, no deployment is initiated until the error is fixed. Visitors of the build site are informed about the state of the build and are pointed to potential errors.
Our CI system is available to any ontology registered with the OBO library. So far we are focusing on ontologies that are heavily interlinked with other ontologies through logical axioms, such as GO, the Human Phenotype Ontology and Uberon.
For collections of interdependent ontologies there is a need not only to automatically test ontologies in isolation, but also to perform integration tests. If this integration is continuous rather than delayed until major releases then ontology developers receive immediate feedback on the changes they make.
We have implemented an ontology continuous integration system that makes use of OWL reasoners and custom tests. To our knowledge, this is the first application of an off-the-shelf continuous integration system applied to managing multiple interconnected ontologies. The system is currently being used successfully in the development of the HPO and the GO, and is being expanded to other ontologies in the OBO Library.
We thank the GO editors for valuable feedback on the Jenkins implementation.
SB conceived of the idea and maintains the Charit/HPO Hudson instance. AI wrote the perl software for checking obo files. HUD wrote Oort. HUD, SC and CJM maintain the Berkeley Jenkins environment. CJM authored the manuscript.
 M. Ashburner, C. A. Ball, J. A. Blake, D. Botstein, H. Butler, J. M. Cherry, A. P. Davis, K. Dolinski, S. S. Dwight, J. T. Eppig, M. A. Harris, D. P. Hill, L. Issel-Tarver, A. Kasarskis, S. Lewis, J. C. Matese, J. E. Richardson, M. Ringwald, G. M. Rubin, and G. Sherlock. Gene ontology: tool for the unification of biology. The Gene Ontology Consortium. Nat Genet, 25(1):25–29, May 2000.
 P. De Matos, R. Alc´antara, A. Dekker, M. Ennis, J. Hastings, K. Haug, I. Spiteri, S. Turner, and C. Steinbeck. Chemical entities of biological interest: an update. Nucleic acids research, 38(suppl 1):D249–D254, 2010.
 J. Deegan, E. Dimmer, and C.J. Mungall. Formalization of taxon-based constraints to detect inconsistencies in annotation and ontology development. BMC bioinformatics, 11(1):530, 2010.
 K. Degtyarenko, P. Matos, M. Ennis, J. Hastings, M. Zbinden, A. McNaught, R. Alcantara, M. Darsow, M. Guedj, and M. Ashburner. ChEBI: a database and ontology for chemical entities of biological interest. Nucleic acids research, 2007.
 Heiko Dietze, David Osumi-Sutherland, Amelia Ireland, and Christopher J Mungall. The obo ontology release tool. 2012.
 M. Fowler and M. Foemmel. Continuous integration. 2006.
 M. Horridge. The OWL API: A Java API for Working with OWL 2 Ontologies. In 6th OWL Experiences and Directions Workshop (OWLED 2009), 2009.
 M. Horridge, D. Tsarkov, and T. Redmond. Supporting early adoption of OWL 1.1 with Protege-OWL and FaCT++. In Proc. of the OWL Experiences and Directions Workshop (OWLED06) at the ISWC, volume 6, 2006.
 S. Koehler, M.H. Schulz, P. Krawitz, S. Bauer, S. Doelken, C.E. Ott, C. Mundlos, D. Horn, S. Mundlos, and P.N. Robinson. Clinical Diagnostics in Human Genetics with Semantic Similarity Searches in Ontologies. The American Journal of Human Genetics, 85(4):457–464, 2009.
 Sebastian Kohler, Sebastian Bauer, Chris Mungall, Gabriele Carletti, Cynthia Smith, Paul Schofield, George Gkoutos, and Peter Robinson. Improving ontologies by automatic reasoning and evaluation of logical definitions. BMC Bioinformatics, 12(1):418, 2011.
 Terrence Meehan, Anna Maria Masci, Amina Abdulla, Lindsay Cowell, Judith Blake, Christopher Mungall, and Alexander Diehl. Logical development of the cell ontology. BMC Bioinformatics, 12(1):6, 2011.
 Christopher Mungall, Georgios Gkoutos, Cynthia Smith, Melissa Haendel, Suzanna Lewis, and Michael Ashburner. Integrating phenotype ontologies across multiple species. Genome Biology, 11(1):R2, 2010.
 Christopher J. Mungall, Michael Bada, Tanya Z. Berardini, Jennifer Deegan, Amelia Ireland, Midori A. Harris, David P. Hill, and Jane Lomax. Cross-Product Extensions of the Gene Ontology. Journal of Biomedical Informatics, 44(1):80 – 86, 2011. Ontologies for Clinical and Translational Research.
 C.J. Mungall, A. Ruttenberg, and D. Osumi-Sutherland. Taking shortcuts with OWL using safe macros. 2011.
 C.J. Mungall, C. Torniai, G.V. Gkoutos, S.E. Lewis, and M.A. Haendel. Uberon, an integrative multi-species anatomy ontology. Genome Biology, 13(1):R5, 2012.
 Alan L. Rector. Modularisation of domain ontologies implemented in description logics and related formalisms including OWL. In Proceedings of the 2nd international conference on Knowledge capture, pages 121–128, Sanibel Island, FL, USA, 2003. ACM.
 P.N. Robinson and S. Bauer. Introduction to Bio-Ontologies. Chapman & Hall/Crc Mathematical and Computational Biology. Taylor and Francis, 2011.
 P.N. Robinson, S. Koehler, S. Bauer, D. Seelow, D. Horn, and S. Mundlos. The Human Phenotype Ontology: a tool for annotating and analyzing human hereditary disease. The American Journal of Human Genetics, 83(5):610–615, 2008.
 R. Shearer, B. Motik, and I. Horrocks. HermiT: A Highly- Efficient OWL Reasoner. In Proceedings of the 5th International Workshop on OWL: Experiences and Directions (OWLED 2008), 2008.
 Barry Smith, Michael Ashburner, Cornelius Rosse, Jonathan Bard, William Bug, Werner Ceusters, Louis J Goldberg, Karen Eilbeck, Amelia Ireland, Christopher J Mungall, The OBI Consortium, Neocles Leontis, Philippe Rocca-Serra, Alan Ruttenberg, Susanna-Assunta Sansone, Richard H Scheuermann, Nigam Shah, Patricia L Whetzel, and Suzanna Lewis. The OBO Foundry: coordinated evolution of ontologies to support biomedical data integration. Nat Biotechnol, 25(11):1251–1255, Nov 2007.
 C. J. Wroe, R. Stevens, C. A. Goble, and M. Ashburner. A methodology to migrate the gene ontology to a description logic environment using DAML+OIL. Pac Symp Biocomput, pages 624–35, 2003. Journal Article.
*to whom correspondence should be addressed
2 http://jenkins-ci.org/ Jenkins and Hudson were originally the same piece of software, but the codebase at one point forked. From here on we refer specifically to Jenkins
6 See http://douroucouli.wordpress.com/2012/04/24/taxon-constraints-inowl for details