From the Language Guide:
AppleScript provides the text item delimiters property for use in processing text. This property consists of a list of strings used as delimiters by AppleScript when it coerces a list to text or gets text items from text strings. When getting text items of text, all of the strings are used as separators. When coercing a list to text, the first item is used as a separator.
In essence, you can set a delimiter (or a list of them) and then split or separate a string on it, creating a list of the resulting text items.
In this scenario, the scripts are splitting the text on the chevron character and creating a list of the resulting strings.
You can also rejoin items in a list around the delimiter by coercing the list with as text. That does not occur in this scenario since the purpose of the split is simply to get the length of each string rather than to alter it.
Here is a simple example to demonstrate changing every instance of the left chevron to a right chevron.
set str to "first second « third fourth « fifth sixth"
set text item delimiters to "«"
set strList to text items of str
--> {"first second ", " third fourth ", " fifth sixth"}
set text item delimiters to "»"
set str to strList as text
--> "first second » third fourth » fifth sixth"
An index is simply the position of an ordered element within its grouping. For example, the letter ‘e’ is the 5th character of the string ‘apple’, so its index is 5.
character 5 of “apple”
→ e
You can read more about index here: Reference forms