Name, Job, Bye.
A dialogue systems is the way the player talk to other agents in the game.
In this form of interaction the player only choice is to to talk to a agent. The agent then delivers the lines and the conversation ends. Talking to the same agent again may make it say other things or it can say things depending on events in the world. They player have however no control over the conversation when it takes place. The conversation is often in a form of a small cut-scene to be able to show the faces of both the player and the agent as they speak. One common variation is the silent protagonist, that is the player character do not speak at all.
Dialogue trees or Branching
With a dialogue tree the agent say something and then the player get a list of possible responses where each response is a line of text of what the player will say. When one is selected the agents response is shown and then the player get a new list of responses. So the things that can be said branch out like a tree from the start line. It is common to have multiple responses lead back on the same path to limit the amount of dialogue that must be produced. These section that merge again can still matter as they can modify other things such as what the agent thinks about the player.
A option for the dialogue trees is to use a Hub-and-Spokes system. When talking to an agent the player can pick a list of things from the main hub of the conversation. Picking a topic the player will then return to back to the main hub when or go to another hub with more options. As it is often possible to return to the main hub the player are free to explore all topics in the order he wish. Sections can still be in normal tree form and closed of when the player visit them to create the common tree style.
Another option for dialogue is a emote driven system. Here to we show short sentance that contains the core of what the player will say. The player then select it and the player character says his full line. The goal is to avoid the player having to read the whole line. To add extra context to the player one can show a symbol with when the player is about to select an option.
In a keyword system the player have a bunch of keywords that he can ask NPC's about. If they know nothing he gets a standard response and if he gets a real answer he might learn more keywords to use. Some system also allow the player to add his own keywords to the list.
This system let the player enter text and then use a text parser to try to find out what to say back. The response can be pre-set or built based around the words used by the player. Not that common in games today.
The Poor Man's Dialogue Tree - 2014
RPG Dialogue Systems - 2013
Reinventing Dialogue - 2009
Branching Conversation Systems and the Working Writer - 2009
RPG dialogue systems - 2008
Defining Dialogue Systems - 2008