(27th-Nov-2020)
• Deep learning approaches have been very successful in language modeling, machine translation and natural language processing due to the use of embeddings for symbols ( , ) and words ( Rumelhart et al. 1986a Deerwester 1990 Bengio et al., ; et al., 2001). These embeddings represent semantic knowledge about individual words and concepts. A research frontier is to develop embeddings for phrases and for relations between words and facts. Search engines already use machine learning for this purpose but much more remains to be done to improve these more advanced representations.
• One interesting research direction is determining how distributed representations can be trained to capture the relations between two entities. These relations allow us to formalize facts about objects and how objects interact with each other. In mathematics, a binary relation is a set of ordered pairs of objects. Pairs that are in the set are said to have the relation while those who are not in the set do not. For example, we can define the relation “is less than” on the set of entities {1,2,3} by defining the set of ordered pairs S ={(1,2),(1, 3), (2,3)}. Once this relation is defined, we can use it like a verb. Because (1,2)∈ S, we say that 1 is less than 2. Because (2,1) ∈ S, we can not say that 2 is less than 1. Of course, the entities that are related to one another need not be numbers. We could define a relation containing tuples like ( , ). is_a_type_of dog mammal In the context of AI, we think of a relation as a sentence in a syntactically simple and highly structured language. The relation plays the role of a verb, while two arguments to the relation play the role of its subject and object. These sentences take the form of a triplet of tokens.
• For example, we could define the has_fur attribute, and apply it to entities like dog. Many applications require representing relations and reasoning about them. How should we best do this within the context of neural networks? Machine learning models of course require training data. We can infer relations between entities from training datasets consisting of unstructured natural language. There are also structured databases that identify relations explicitly. A common structure for these databases is the relational database, which stores this same kind of information, albeit not formatted as three token sentences. When a database is intended to convey commonsense knowledge about everyday life or expert knowledge about an application area to an artificial intelligence system, we call the database a knowledge base. Knowledge bases range from general ones like Freebase, OpenCyc, WordNet, or Wikibase,1 etc. to more specialized knowledge bases, like GeneOntology.2 Representations for entities and relations can be learned by considering each triplet in a knowledge base as a training example and maximizing a training objective that captures their joint distribution (Bordes et al., ).
Comments