Article 2: The Doc Object, Sentence Detection, and Tokens
Last time we got spaCy installed and ran our first sentence through it, printing out the part of speech and lemma for each token without really stopping to ask what we'd actually been handed back. That's where this article picks up: what a Doc object really is, how spaCy figures out where one sentence ends and the next begins, and what "tokenization" is actually doing under the hood.
The Doc Object, Properly This Time
When you call nlp(text), what comes back isn't a string and it isn't a list - it's a Doc, and the easiest way to think about it is as a sequence of Token objects. Like a Python list, you can take its length, index into it, and slice it:
import spacy
nlp = spacy.load("en_core_web_sm")
doc = nlp("Desk2Mob is a blog about building things with AI and cloud tools.")
print(len(doc))
print(doc[0])
print(doc[1:4])
print(type(doc[1:4]))
13
Desk2Mob
is a blog
<class 'spacy.tokens.span.Span'>
A few things worth noticing. The sentence has 13 tokens, not 12 - the trailing period counts as its own token, which is easy to forget until you're counting on it. doc[0] gives you back a single Token. But slicing, doc[1:4], doesn't give you a list of tokens - it gives you a Span, a lightweight view into a stretch of the original Doc. You'll see Span again and again in spaCy: sentences, matched phrases, and named entities are all represented as spans over the same underlying Doc, rather than as copies of the text.
Sentence Detection
Splitting text into sentences sounds like a job for string.split(". "), until you remember how many periods in English don't actually end a sentence - abbreviations, initials, decimal numbers, and titles like "Dr." or "Mr." all use periods without ending anything. spaCy handles this with doc.sents. In the default en_core_web_sm pipeline, it works out the sentence boundaries from the trained dependency parser instead of a fixed list of rules, which is why it copes with the cases a naive split on periods would get wrong.
One quick note before we go further: the examples from here on reuse the nlp object we loaded in the first code block. If you're copying a snippet on its own, add import spacy and nlp = spacy.load("en_core_web_sm") at the top, otherwise you'll hit a NameError.
text = "Desk2Mob covers AI, cloud, and software engineering. Dr. Patel wrote a great post on NLP last month. It's worth a read."
doc = nlp(text)
for sent in doc.sents:
print(sent.text)
Desk2Mob covers AI, cloud, and software engineering.
Dr. Patel wrote a great post on NLP last month.
It's worth a read.
Three sentences, correctly split, and "Dr." didn't trip it up. And as you'd expect by now, each sentence that doc.sents yields is a Span - one of those same views over the underlying Doc.
Tokens in spaCy
Tokenization is the step that happens before any of this - breaking raw text into the individual pieces (tokens) that everything else operates on. It's tempting to assume that just means splitting on spaces, but spaCy's tokenizer does more than that: it pulls apart contractions, and it recognizes which periods belong to abbreviations it shouldn't split on.
doc = nlp("I don't think spaCy's tokenizer will split U.K. wrong.")
print([token.text for token in doc])
['I', 'do', "n't", 'think', 'spaCy', "'s", 'tokenizer', 'will', 'split', 'U.K.', 'wrong', '.']
Notice "don't" became "do" and "n't", and "spaCy's" became "spaCy" and "'s" - each split into separate tokens according to spaCy's English tokenization rules, instead of being left as one glued-together word. "U.K.", on the other hand, stayed exactly as it was, because spaCy's tokenizer has a built-in exception list for cases like that, so it doesn't chop it into three tokens at every period. None of this comes from guessing. The tokenizer follows language-specific rules and exception lists that spaCy ships with, and everything else in the pipeline builds on the tokens it produces.
Once you've got tokens, each one carries a handful of boolean flags that are genuinely useful for cleaning up text before you do anything else with it:
doc = nlp("It costs $9.99, which isn't bad at all.")
for token in doc:
print(token.text, token.is_punct, token.is_stop, token.like_num)
It False True False
costs False False False
$ False False False
9.99 False False True
, True False False
which False True False
is False True False
n't False True False
bad False False False
at False True False
all False True False
. True False False
is_punct flags punctuation such as commas and periods, and like_num catches number-like text such as "9.99" even though it isn't a plain integer. is_stop tells you whether a token appears in spaCy's built-in stop-word list - the small, high-frequency words like "it", "which", "is", "at", and "all". It's tempting to strip all of those out before counting words, and for something like a word-frequency table that's often a reasonable move. But be careful: is_stop only means "this word is on the stop list," not "this word is safe to throw away." Look at the output above and you'll see "n't" flagged as a stop word - and that's the very piece carrying the negation in "isn't bad". Drop it automatically and "isn't bad" quietly becomes "bad". Whether you remove stop words really depends on what you're trying to analyze.
That's where we're headed next: a closer look at stop words and lemmatization, then putting them to work in a simple word-frequency example.

