I create this effect in photoshop cs3 and it’s work with video too.

I use:

1. Duplication of current Layer ( i named it “dup” )
2. Invert dup (ctrl-i)
3. Set dup layer to Color Dodge
4. Convert to Smart Object (important for video and to change settings)
5. Apply Gaussian Blur (value:3.5 but your can change it)
6. Appy Threshold (value:235 you can change)

Now you can touch the Gaussian Blur to fill better edge lines and Threeshold value for better image clear.
Thanks to Smart Object you can edit this value and u can see effect in preview mode

You can download action file here

If you want a colored cartoon than you can use:

1. Duplication of current Layer ( i named it “dup” )
2. Set dup layer to Color Dodge
3. Convert to Smart Object (important for video and to change settings)
4. apply Poster Edges Filter (value :thickness 5, intensity:4 , Posterization:2)

Change value of Poster Edges for better results.

example:
[youtube -xPbkN4zC5o]