Introduction to Natural Language Processing
Project: The Total Library

Information overload
It is great to have access to huge amounts of information, but since we are not reading faster than before, we can not take advantage of this new situation. Therefore the need of a discipline that help human beings deal with all that data is fundamental.

Natural language processing is the process of building computational models for understanding natural language. It studies the problems of automated generation and understanding of natural human languages. NLP includes natural-language-generation systems that convert information from computer databases into normal human language and natural-language-understanding systems that convert samples of human language into more formal representations that are easier for computer programs to manipulate.

NLP also studies the information contained in human generated texts, along with its language structure.NLP is a multidisciplinary field, which studies artificial intelligence techniques, multivariate statistics, linguistics and any other domain that can be used to process, generate or interpret language with computers.

NLP Facts
-The Turing test is a proposal for a test of a machine's ability to demonstrate intelligence. Described by Alan Turing in the 1950 paper "Computing Machinery and Intelligence," it proceeds as follows: a human judge engages in a natural language conversation with one human and one machine, each of which tries to appear human. All participants are placed in isolated locations. If the judge cannot reliably tell the machine from the human, the machine is said to have passed the test. In order to test the machine's intelligence rather than its ability to render words into audio, the conversation is limited to a text-only channel such as a computer keyboard and screen. This test was the first mainstream experiment related to NLP.
-Text is the largest repository of human knowledge and is growing quickly, there are emails, news articles, web pages, chat archives, scientific articles, insurance claims, customer complaints letters, transcripts of phone calls, technical documents, government documents, patent portfolios, court decisions, contracts, and so on.

-Nowadays we have access to huge amounts of information, much more than in the past decades, one of the problems with this is that we are not reading any faster than before, therefore we can not take full advantage of this new situation. NlP tries to optimize the human usage of information.

-Dealing with natural language is a difficult task. We need to understanding multiple disciplines including multivariate statistics, learning algorithms, clustering, hidden Markov models and part of speech tagging. We need to have knowledge about language, grammar, ontology and folksonomy.

-Processing of a huge amount of data in a limited amount of time is required so special algorithms are needed. We generally apply algorithms that have low computational cost or algorithms that allow reducing the amount of computational processing needed by pre-processing the data we have. To do this there are techniques for reducing the size of the text by extracting stop words, removing words that appear too often and also words that appear very few times.

-The applications of NLP include answering queries, identifying spam, recognizing what is the main theme of a document, grouping similar texts, obtaining the main keywords of a document, detecting syntactic errors and identifying the secondary themes of a document.

Sun, Feb 21, 2010  Permanent link

Sent to project: The Total Library

RSS for this post