While it may seem annoying that a CSV can in fact be separated by something
other than a comma, it is actually very convenient. Say, for example, your data contains commas in places that are
not supposed to be used as separators (addresses for example, or currency, etc.). This can be confusing to the computer program trying to read your data - and is in fact why
quotation marks are used as text qualifiers, to help solve this problem. Instead of using commas both within your text fields and as a delimiter, you can opt to use another delimiter - one that is not used elsewhere within your data.
You can see this more practically when looking at the delimiter defaults by locale. For example, in Europe it is much more common to see the default delimiter as a semi-colon, because commas are used to separate decimals (in currency, for example).
To bring us back to the core concept:
The delimiter in your CSV is the character (comma or otherwise) that separates the data in your file into distinct fields. Practically speaking, it is what allows you to open your file in a spreadsheet and view the data in nicely organized columns and rows, and even what allows programs to import data from your CSV and place the data into the correct fields in your database. Delimiters are the key to the entire structure of CSV!