Tokenizer: fun with strings and dynamic memory allocation

Starter code: tokenizer-package.tar
Upload via Moodle as: tokenizer.tar

Goals

• Get used to the syntax of pointers
• Practice manipulating null-terminated strings
• Practice using malloc and free
• Practice checking the return values of system functions

Rubric

Note that "correctness" in this rubric includes "meets all documented specifications". It also means that you need to check return values (especially of malloc) to ensure the integrity of your data structures.

Your assignment

• Implement the functions documented in the tokenizer.h file in tokenizer-package.tar.
• Put your implementations in tokenizer.c (a starter version of which is provided in the package).
• Do not change tokenizer.h.
• Submit your file(s) as tokenizer.tar via Moodle.

The only file you are required to submit is tokenizer.c. You may include your own main.c and Makefile, but we will use our own main.c and Makefile to do our testing and grading.

A little advice

• Read the function definitions in tokenizer.h carefully.
• Do it right away, even if you don't have time to start coding quite yet.
• Start with get_token_count. You can test it separately (though I have not included an explicit test for it in main.c), and you may find it useful when you get to implementing get_tokens.

Have fun!