Tokenizer: fun with strings and dynamic memory allocation

Starter code: tokenizer-package.tar
Upload via Moodle as: tokenizer.tar

Goals

Get used to the syntax of pointers
Practice manipulating null-terminated strings
Practice using malloc and free
Practice checking the return values of system functions

Rubric

3 - get_token_count correctness 5 - get_tokens correctness 4 - free_tokens correctness 3 - code quality

Note that "correctness" in this rubric includes "meets all documented specifications". It also means that you need to check return values (especially of malloc) to ensure the integrity of your data structures.

Your assignment

Implement the functions documented in the tokenizer.h file in tokenizer-package.tar.
Put your implementations in tokenizer.c (a starter version of which is provided in the package).
Do not change tokenizer.h.
Submit your file(s) as tokenizer.tar via Moodle.

The only file you are required to submit is tokenizer.c. You may include your own main.c and Makefile, but we will use our own main.c and Makefile to do our testing and grading.



        A little advice
        
        Read the function definitions in tokenizer.h carefully.
        Do it right away, even if you don't have time to start coding quite yet.
        Start with get_token_count. You can test it separately (though I
        have not included an explicit test for it in main.c), and you may
        find it useful when you get to implementing get_tokens.
        

        Have fun!