Tokens

Tokens in programming languages are the smallest individual units of code that have a meaning to the compiler or interpreter. They serve as the building blocks of a program and are classified into several categories, each contributing to the syntax and semantics of the language.

---

Some common token types are -

1. Keywords

In programming languages keywords are thr reserved words that have predefined meanings and cannot be used as identifiers for variables, functions, or other entities. It have special meanings to the compiler or interpreter.

Example - and or not if else as in import

---

2. Identifiers

User-defined names for variables, functions, arrays and user-defined elements.

Example - user_name, address, location, email

---

3. Constants

Fixed values that cannot be altered during program execution. Constants can be defined using the const keyword or the #define preprocessor directive.

---

4. Literals

• String literals : String literals are sequences of characters enclosed in quotes. They represent textual data and can be used in various contexts, such as displaying text on the screen or concatenating strings. In many languages, string literals can be either single-quoted (') or double-quoted ("), although the exact rules vary between languages.
  example - "John", "smith", "alice", "bob"
• Numeric Literals : Numeric literals represent numerical values, including integers and floating-point numbers. They can be expressed in various forms, such as decimal, octal, hexadecimal, or binary, depending on the language and the context in which they are used.
  example - 1 1.7 0xf1 85 0

---

5. Operators

Operators are symbols that perform operations on one or more operands.

example - + - * / ^ ~ > < >= <= & || ! =

---

6. Punctuation Marks

Punctuation mark are used to separate tokens, indicate the boundaries of statements, and control the order of operations.

example - , ; ( { [ ] } )(Note - example may vary and depends on language to language )