wAIpoint
Italiano

Grammar & parsing

Level: Intermediate

What it is

A grammar is a set of rules saying how words combine into phrases and sentences. Parsing is the job of reading a sentence and finding its structure: which words form a noun phrase, what the verb refers to. Many ambiguities ("I saw the man with the telescope") are ambiguities of structure.

What it is for

Understanding the structure of sentences in tools that need exact meaning, such as programming-language compilers, query languages, voice commands and information extraction. Neural language models usually learn such structure implicitly, but explicit parsing is still useful when you need rules you can inspect.

How to do it

The process, step by step. Libraries and tools change; these steps stay.

  1. Decide what you need: a full sentence tree, or just relations such as who did what to whom.
  2. Cut the text into sentences and words (tokenisation) and tag each word's role (noun, verb, ...).
  3. Run a parser from a language-processing library, which gives a tree or a set of word-to-word links for each sentence.
  4. Read the output on a few real sentences, including awkward ones, to see where it fails.
  5. For a small, fixed language (commands, formulas), write the grammar yourself and let a parser generator handle it.
  6. Use the structure for your task, for example to pull out the subject and object of every sentence.

Timeline

  1. 1956Chomsky: three models for the description of language. source

Sources