You should be able to. The quantity of milk added to tea is so small that, for most people, it will barely register as a carb or a fat. The people who have to avoid it are those with milk intolerance, but I have actually come across few people with milk intolerance. Don't forget normal tea has caffe...
