Entity Linking via Topic Models in Apache Spark
Name
Olha Kaminska
Abstract
Entity linking is a field of natural language processing that aims to define the real meaning of a word in a particular text. The same term can have different meanings in different contexts, which demonstrates the importance of the field. Entity linking is actively applied to real-world business problems. One widely known problem is defining companies with similar products to investigate competitors on the market. In this task, products represent entities, and the target of the entity linking is to connect the same or similar products among an assortment of different companies.
In the current work, similar products from different Estonian companies are linked based on their textual descriptions. In the obtained results, every company is linked with at least one other company through similar products.
To define similar products, the textual descriptions are divided into clusters using four different topic modeling techniques. Based on the obtained clusters, linked graphs are built in the Apache Spark environment and manual tests and comparisons using statistical measures are performed. The graphs based on latent Dirichlet allocation topic modeling approaches show the best results.
The performance of the methods illustrates that topic modeling techniques can be used for entity linking and can provide practical results.
In the current work, similar products from different Estonian companies are linked based on their textual descriptions. In the obtained results, every company is linked with at least one other company through similar products.
To define similar products, the textual descriptions are divided into clusters using four different topic modeling techniques. Based on the obtained clusters, linked graphs are built in the Apache Spark environment and manual tests and comparisons using statistical measures are performed. The graphs based on latent Dirichlet allocation topic modeling approaches show the best results.
The performance of the methods illustrates that topic modeling techniques can be used for entity linking and can provide practical results.
Graduation Thesis language
English
Graduation Thesis type
Master - Computer Science
Supervisor(s)
Pelle Jakovits, Peep Küngas
Defence year
2019