Measuring Conceptual Distance: The Design of a Metric for Measuring the Semantic Similarity of Word Substitution Pairs

William Lewis, Merrill Garrett, Jason Barker
wlewis@u.arizona.edu, garrett@u.arizona.edu, jbarker@u.arizona.edu
University of Arizona

Word substitution errors come in two basic flavors: (1) semantically related and (2) phonologically related (with occasional mixes between the two):

(1)

"It's a single four-year sentence..."
(target was: 'four-year term')

(2)

"People don't know! - people who've never sat on an envelope!"
(target was 'elephant')

Of the two, semantically related are the more common, but have been more difficult to quantify. Fay and Cutler 1977 first proposed a structural metric for measuring the phonological similarity of target and intrusion in word substitutions, and they noted a strong dissociation in phonological similarity and degree of semantic relation in word substitution pairs. Semantic relation was assessed intuitively in their comparison, which is typical of subsequent research.

A metric for measuring semantic similarity should show graded similarity between word pairs, and should allow measurements across a large vocabulary. We propose using WordNet, a semantic network containing 70,000+ entries (see Miller 1995), as a tool for calculating such measurements. A common approach for measuring semantic similarity in a semantic network is to count the number of nodes between two entries in the network (Rada et al 1989). The problem is that metrics of this type fail to take into account the fluctuating density across a network. At least two methods for adjusting similarity metrics have been proposed: (a) adjustments based on depth (Richardson et al 1994), or (b) adjustments based on the local density relative to a region within the network (Agirre and Rigau 1996). Although depth is correlated with density within WordNet, (a) fails because the correlation is weak. Adjustments based on regional density (b) depend on how one carves up the semantic space, and work best when measuring distance relative to the region selected; one would hope for an absolute measure.

We have implemented a semantic similarity measure using the simple algorithm proposed by Rada et al 1989, but adjusted using averaged local density for the terms being compared (local density being calculated by counting the number of nodes in the immediate neighborhood of a particular node). Preliminary results from running the algorithm against a large number of semantic and phonological speech error pairs have been promising. Using a scale of similarity from zero to one, with zero indicating identity, and one indicating no relation, nearly all semantic pairs (84%) score below 0.2. Random pairings of the same terms tend to measure above 0.5, and the phonological pairings show strong dissociation with the measure (generally having no connection, thus resolving to 1.0).

References

Agirre, Eneko and German Rigau. "Word Sense Disambiguation Using Conceptual Density." In: Proceedings of the 16th International Conference on Computational Linguistics, Copenhagen, 1996.

Fay, D. and Cutler, A. (1977) Malapropisms and the structure of the mental lexicon. Linguistic Inquiry, 8, 505-520.

Miller, George A. (1995). WordNet: a lexical database for English. Communications of the ACM 38 (11), 39 - 41.

Rada, Roy, Hafedh Mili, Ellen Bicknell, and Maria Blettner (1989). Development and Application of a Metric on Semantic Nets. IEEE Transactions on Systems, Man and Cybernetics 19: 17-30.

Richardson, R., A. Smeaton, and J. Murphy (1994). Using WordNet as a Knowledge Base for Measuring Semantic Similarity between Words. Technical Report, Working paper CA-1294, School of Computer Applications, Dublin City University: Dublin, Ireland.